From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: bug#47336: Disarchive as a fallback for downloads Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 04:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16164745447069 (code B ref -1); Tue, 23 Mar 2021 04:43:01 +0000 Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000 Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs3-0001pw-WB for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:51240) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs2-0001pp-0u for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59600) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYs1-0007wV-QR for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYry-0007rZ-7A for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id D60141373 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/ YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 00:42:12 -0400 Message-ID: <87eeg6o50b.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@ngyro.com; helo=wout3-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.6 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) Hello, This patch series adds Disarchive assembly (backed by SWH lookup) as a fallback for downloads. To try it, make sure you are running the daemon in an environment with Disarchive available: $ ./pre-inst-env guix environment --ad-hoc guile disarchive # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild Don=E2=80=99t forget to stop your existing Guix Daemon. :) You also need to make sure that regular downloads are unavailable. I do this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl= -fetch=E2=80=9D in =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of URL= s with =E2=80=98()=E2=80=99: (let try ((uri (append uri content-addressed-uris))) (match '() ; uri ...)) Now you can ask Guix for a recent .tar.gz source package: $ ./pre-inst-env guix build --no-substitutes -S python-httpretty You should see: Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v= diwlv19y9-httpretty-1.0.5.tar.gz Assembling the directory httpretty-1.0.5 Downloading from Software Heritage... 7903d608efc89c14afb4d692a3721156e31a43e2/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING [...] Checking httpretty-1.0.5 digest... ok Assembling the tarball httpretty-1.0.5.tar Checking httpretty-1.0.5.tar digest... ok Assembling the Gzip file httpretty-1.0.5.tar.gz Checking httpretty-1.0.5.tar.gz digest... ok Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty= -1.0.5.tar.gz successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett= y-1.0.5.tar.gz.drv There=E2=80=99s lots to talk about though.... First, it looks up the metadata on my server. This is fine for a demo, but not what we want forever. The patch series supports adding several mirrors for looking up the metadata. In the past, we talked about putting everything on one or a few of the big Git hosting platforms like GitHub or Gitlab. That way, it would be easily picked up by SWH and archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to bu= ild the metadata, and a little script that moves it from the build server to my Web server. It would be simple enough to adjust that script to push it to a remote Git repo. (Of course, the next step is to move this setup to Guix infrastructure.) Thoughts? On the code level, there were two things I couldn=E2=80=99t figure out for myself. I made the mirror list just simple strings. AIUI, the client and the daemon have to agree about the format of the mirror list. Given that running old daemons is common, changing the format is difficult. Is it worth it to copy the more flexible interface used by the content addressed mirrors? If yes, do I have to do the same =E2=80=98module-autolo= ad!=E2=80=99 dance to use =E2=80=98bytevector->base16-string=E2=80=99? :) (I probably = would have just copied it, but that part confused me a bit.) I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel= l, just =E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D). It feels weird to use= a bunch of host-side modules from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module. This = is okay because =E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. = It=E2=80=99s more like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2= =80=9D, which is why importing host-side modules is OK... right? Hopefully everything else is more-or-less fine. :) -- Tim From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: bug#47336: [PATCH 1/2] swh: Add a directory download procedure. References: <87eeg6o50b.fsf@ngyro.com> In-Reply-To: <87eeg6o50b.fsf@ngyro.com> Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 04:53:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org Cc: Timothy Sample Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.16164751648063 (code B ref 47336); Tue, 23 Mar 2021 04:53:02 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 04:52:44 +0000 Received: from localhost ([127.0.0.1]:59255 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ24-00025z-Ds for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:52:44 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:42087) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ23-00025o-5S for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 00:52:43 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 5C2E14CE; Tue, 23 Mar 2021 00:52:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:52:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=9GhkL9aQDX+o8K1Ib LLy06+cd1v9DpMcsGDadXOnnPk=; b=mD0rvrRxnVCL73RGYUbbdN8818yUnGh58 WE9F+dIqu3kz3MZxL1mHEkGX5ba3xkVlxU0Jk++lJ93qngoCNF5t9lxuMudGhDgG YaIBxdv+jeNcLz36AbuwJgtQCT4he/E9bBtXImaqdWFyLyJ8DNCZb6c+j9wX/c2g Ys/CmVIQkVKME2MiVs1SdTS3AL9gzTsGMAow2k9UkdKWb4RKX0AT7QziTP8bokdi a6wpQbxcNE2qz1LgmAZX195fBJt4JbWUNUgyxIpJtMMixqgBz7SPR+lhrAQxolMh oU+h6LKZRvuT6cOaXWFNYCE3s59m1uWyFi6I1CsN1qsafElYVts0g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgggfestdekredtre dttdenucfhrhhomhepvfhimhhothhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhg hihrohdrtghomheqnecuggftrfgrthhtvghrnhepvdehudduieektedugeevgfeutdffff dujeeifffhudehvdelveduvedtfefhtdejnecukfhppeejgedrudduiedrudekiedrgeeg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshgrmh hplhgvthesnhhghihrohdrtghomh X-ME-Proxy: Received: from mrblack.lan (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 738071080054; Tue, 23 Mar 2021 00:52:36 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 00:52:12 -0400 Message-Id: <20210323045213.9419-1-samplet@ngyro.com> X-Mailer: git-send-email 2.31.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/swh.scm (swh-directory-download): New procedure (with implementation extracted from 'swh-download'). (swh-download): Use it to download the revision directory. --- guix/swh.scm | 65 +++++++++++++++++++++++++++++----------------------- 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/guix/swh.scm b/guix/swh.scm index f11b7ea2d5..2402ec98e6 100644 --- a/guix/swh.scm +++ b/guix/swh.scm @@ -108,6 +108,7 @@ commit-id? + swh-download-directory swh-download)) ;;; Commentary: @@ -558,12 +559,6 @@ requested bundle cooking, waiting for completion...~%")) ;;; High-level interface. ;;; -(define (commit-id? reference) - "Return true if REFERENCE is likely a commit ID, false otherwise---e.g., if -it is a tag name. This is based on a simple heuristic so use with care!" - (and (= (string-length reference) 40) - (string-every char-set:hex-digit reference))) - (define (call-with-temporary-directory proc) ;FIXME: factorize "Call PROC with a name of a temporary directory; close the directory and delete it when leaving the dynamic extent of this call." @@ -577,6 +572,39 @@ delete it when leaving the dynamic extent of this call." (lambda () (false-if-exception (delete-file-recursively tmp-dir)))))) +(define* (swh-download-directory id output + #:key (log-port (current-error-port))) + "Download from Software Heritage the directory with the given ID, and +unpack it to OUTPUT. Return #t on success and #f on failure" + (call-with-temporary-directory + (lambda (directory) + (match (vault-fetch id 'directory #:log-port log-port) + (#f + (format log-port + "SWH: directory ~a could not be fetched from the vault~%" + id) + #f) + ((? port? input) + (let ((tar (open-pipe* OPEN_WRITE "tar" "-C" directory "-xzvf" "-"))) + (dump-port input tar) + (close-port input) + (let ((status (close-pipe tar))) + (unless (zero? status) + (error "tar extraction failure" status))) + + (match (scandir directory) + (("." ".." sub-directory) + (copy-recursively (string-append directory "/" sub-directory) + output + #:log (%make-void-port "w")) + #t)))))))) + +(define (commit-id? reference) + "Return true if REFERENCE is likely a commit ID, false otherwise---e.g., if +it is a tag name. This is based on a simple heuristic so use with care!" + (and (= (string-length reference) 40) + (string-every char-set:hex-digit reference))) + (define* (swh-download url reference output #:key (log-port (current-error-port))) "Download from Software Heritage a checkout of the Git tag or commit @@ -593,28 +621,7 @@ wait until it becomes available, which could take several minutes." (format log-port "SWH: found revision ~a with directory at '~a'~%" (revision-id revision) (swh-url (revision-directory-url revision))) - (call-with-temporary-directory - (lambda (directory) - (match (vault-fetch (revision-directory revision) 'directory - #:log-port log-port) - (#f - (format log-port - "SWH: directory ~a could not be fetched from the vault~%" - (revision-directory revision)) - #f) - ((? port? input) - (let ((tar (open-pipe* OPEN_WRITE "tar" "-C" directory "-xzvf" "-"))) - (dump-port input tar) - (close-port input) - (let ((status (close-pipe tar))) - (unless (zero? status) - (error "tar extraction failure" status))) - - (match (scandir directory) - (("." ".." sub-directory) - (copy-recursively (string-append directory "/" sub-directory) - output - #:log (%make-void-port "w")) - #t)))))))) + (swh-download-directory (revision-directory revision) output + #:log-port log-port)) (#f #f))) -- 2.31.0 From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: bug#47336: [PATCH 2/2] download: Use Disarchive as a last resort. Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 04:54:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org Cc: Timothy Sample Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.16164752218171 (code B ref 47336); Tue, 23 Mar 2021 04:54:02 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 04:53:41 +0000 Received: from localhost ([127.0.0.1]:59262 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ2y-00027i-R7 for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:53:41 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:51937) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZ2i-000276-Gd for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 00:53:39 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id BCA45140F; Tue, 23 Mar 2021 00:53:18 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:53:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=Sh8vx4Va/ITkI6SHjW/BifjchdPveiaTGxtGoj7SZ yI=; b=rr9U1Tsdg37OXQSTnqsTREm3ajQAetH/38BeY8pNp0b45p9UqZgJKKqZ3 Ee1wD0JjdU39XHa5sgY4qkYehCHXrzwGf0U6EYRD2vCH5B+fMllBeutBXksK68Xn dpxyKXZX7IdwcEYpPfV6MR2u4gvSLsHiVhxsCXBBltlpAEUoS/y2ywNrpV67Mcar rlUGCaM3+eABsrn9ZtG4SVYCtJRCdhpmsictRc8zp16rdDGtBts8acpmXGj05sRO 3zGPzrB11LEl8n6MWVf5PXNksxAtyKsKRZH78r5h4id1jQC99R8tnZ+AL1BpGmkq M9yGmRv/gqrfkAo3U7euPQ2gnK/6g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkofgjfhggtgfgsehtke ertdertdejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvght sehnghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpeehtdeuteejkeevhfeftdeltd etjeejhfettefgleeigedtgfeguddtfffgtdevgeenucffohhmrghinhepnhhghihrohdr tghomhenucfkphepjeegrdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack.lan (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 257551080054; Tue, 23 Mar 2021 00:53:18 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 00:52:13 -0400 Message-Id: <20210323045213.9419-2-samplet@ngyro.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210323045213.9419-1-samplet@ngyro.com> References: <20210323045213.9419-1-samplet@ngyro.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/download.scm (%disarchive-mirrors): New variable. (%disarchive-mirror-file): New variable. (built-in-download): Add 'disarchive-mirrors' keyword argument and pass its value along to the 'builtin:download' derivation. (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'. * guix/scripts/perform-download.scm (perform-download): Read Disarchive mirrors from the environment and pass them to 'url-fetch'. * guix/build/download.scm (disarchive-fetch/any): New procedure. (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to make a list of URIs, and use the new procedure to fetch the file if all other methods fail. --- guix/build/download.scm | 77 +++++++++++++++++++++++++++---- guix/download.scm | 19 ++++++-- guix/scripts/perform-download.scm | 7 ++- 3 files changed, 89 insertions(+), 14 deletions(-) diff --git a/guix/build/download.scm b/guix/build/download.scm index a22d4064ca..f476d0f8ec 100644 --- a/guix/build/download.scm +++ b/guix/build/download.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2015 Mark H Weaver ;;; Copyright © 2017 Tobias Geerinckx-Rice +;;; Copyright © 2021 Timothy Sample ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,10 +24,12 @@ #:use-module (web http) #:use-module ((web client) #:hide (open-socket-for-uri)) #:use-module (web response) + #:use-module (guix base16) #:use-module (guix base64) #:use-module (guix ftp-client) #:use-module (guix build utils) #:use-module (guix progress) + #:use-module (guix swh) #:use-module (rnrs io ports) #:use-module (rnrs bytevectors) #:use-module (srfi srfi-1) @@ -626,10 +629,50 @@ Return a list of URIs." (else (list uri)))) +(define* (disarchive-fetch/any uris file + #:key (timeout 10)) + "Fetch a Disarchive specification from any of URIS, assemble it, +and write the output to FILE." + (define (fetch-specification uris) + (any (lambda (uri) + (false-if-exception* + (let-values (((port size) (http-fetch uri + #:verify-certificate? #t + #:timeout timeout))) + (let ((specification (read port))) + (close-port port) + specification)))) + uris)) + + (define (resolve addresses output) + (any (match-lambda + (('swhid swhid) + (match (string-split swhid #\:) + (("swh" "1" "dir" id) + (format #t "Downloading from Software Heritage...~%" file) + (false-if-exception* + (swh-download-directory id output))) + (_ #f))) + (_ #f)) + addresses)) + + (match (and=> (resolve-module '(disarchive) #:ensure #f) + (lambda (disarchive) + (cons (module-ref disarchive '%disarchive-log-port) + (module-ref disarchive 'disarchive-assemble)))) + (#f #f) + ((%disarchive-log-port . disarchive-assemble) + (format #t "Trying to use Disarchive to assemble ~a~%" file) + (match (fetch-specification uris) + (#f #f) + (spec (parameterize ((%disarchive-log-port (current-output-port))) + (disarchive-assemble spec file #:resolver resolve))))))) + (define* (url-fetch url file #:key (timeout 10) (verify-certificate? #t) (mirrors '()) (content-addressed-mirrors '()) + (disarchive-mirrors '()) (hashes '()) print-build-trace?) "Fetch FILE from URL; URL may be either a single string, or a list of @@ -693,6 +736,17 @@ otherwise simply ignore them." hashes)) content-addressed-mirrors)) + (define disarchive-uris + (append-map (lambda (mirror) + (map (match-lambda + ((hash-algo . hash) + (string->uri + (string-append mirror + (symbol->string hash-algo) "/" + (bytevector->base16-string hash))))) + hashes)) + disarchive-mirrors)) + ;; Make this unbuffered so 'progress-report/file' works as expected. 'line ;; means '\n', not '\r', so it's not appropriate here. (setvbuf (current-output-port) 'none) @@ -705,15 +759,18 @@ otherwise simply ignore them." (or (fetch uri file) (try tail))) (() - (format (current-error-port) "failed to download ~s from ~s~%" - file url) - - ;; Remove FILE in case we made an incomplete download, for example due - ;; to ENOSPC. - (catch 'system-error - (lambda () - (delete-file file)) - (const #f)) - #f)))) + ;; If we are looking for a software archive, one last thing we + ;; can try is to use Disarchive to assemble it. + (or (disarchive-fetch/any disarchive-uris file #:timeout timeout) + (begin + (format (current-error-port) "failed to download ~s from ~s~%" + file url) + ;; Remove FILE in case we made an incomplete download, for + ;; example due to ENOSPC. + (catch 'system-error + (lambda () + (delete-file file)) + (const #f)) + #f)))))) ;;; download.scm ends here diff --git a/guix/download.scm b/guix/download.scm index 30f69c0325..72094e7318 100644 --- a/guix/download.scm +++ b/guix/download.scm @@ -406,12 +406,19 @@ (plain-file "content-addressed-mirrors" (object->string %content-addressed-mirrors))) +(define %disarchive-mirrors + '("https://disarchive.ngyro.com/")) + +(define %disarchive-mirror-file + (plain-file "disarchive-mirrors" (object->string %disarchive-mirrors))) + (define built-in-builders* (store-lift built-in-builders)) (define* (built-in-download file-name url #:key system hash-algo hash mirrors content-addressed-mirrors + disarchive-mirrors executable? (guile 'unused)) "Download FILE-NAME from URL using the built-in 'download' builder. When @@ -422,13 +429,16 @@ explicitly depend on Guile, GnuTLS, etc. Instead, the daemon performs the download by itself using its own dependencies." (mlet %store-monad ((mirrors (lower-object mirrors)) (content-addressed-mirrors - (lower-object content-addressed-mirrors))) + (lower-object content-addressed-mirrors)) + (disarchive-mirrors (lower-object disarchive-mirrors))) (raw-derivation file-name "builtin:download" '() #:system system #:hash-algo hash-algo #:hash hash #:recursive? executable? - #:sources (list mirrors content-addressed-mirrors) + #:sources (list mirrors + content-addressed-mirrors + disarchive-mirrors) ;; Honor the user's proxy and locale settings. #:leaked-env-vars '("http_proxy" "https_proxy" @@ -439,6 +449,7 @@ download by itself using its own dependencies." ("mirrors" . ,mirrors) ("content-addressed-mirrors" . ,content-addressed-mirrors) + ("disarchive-mirrors" . ,disarchive-mirrors) ,@(if executable? '(("executable" . "1")) '())) @@ -492,7 +503,9 @@ name in the store." #:executable? executable? #:mirrors %mirror-file #:content-addressed-mirrors - %content-addressed-mirror-file))))) + %content-addressed-mirror-file + #:disarchive-mirrors + %disarchive-mirror-file))))) (define* (url-fetch/executable url hash-algo hash #:optional name diff --git a/guix/scripts/perform-download.scm b/guix/scripts/perform-download.scm index 8d409092ba..6889bcef79 100644 --- a/guix/scripts/perform-download.scm +++ b/guix/scripts/perform-download.scm @@ -54,7 +54,8 @@ actual output is different from that when we're doing a 'bmCheck' or (output* "out") (executable "executable") (mirrors "mirrors") - (content-addressed-mirrors "content-addressed-mirrors")) + (content-addressed-mirrors "content-addressed-mirrors") + (disarchive-mirrors "disarchive-mirrors")) (unless url (leave (G_ "~a: missing URL~%") (derivation-file-name drv))) @@ -79,6 +80,10 @@ actual output is different from that when we're doing a 'bmCheck' or (lambda (port) (eval (read port) %user-module))) '()) + #:disarchive-mirrors + (if disarchive-mirrors + (call-with-input-file disarchive-mirrors read) + '()) #:hashes `((,algo . ,hash)) ;; Since DRV's output hash is known, X.509 certificate -- 2.31.0 From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: bug#47336: Disarchive as a fallback for downloads References: <87eeg6o50b.fsf@ngyro.com> Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 23 Mar 2021 05:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 47336@debbugs.gnu.org Cc: control@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161647631510444 (code B ref 47336); Tue, 23 Mar 2021 05:12:01 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 05:11:55 +0000 Received: from localhost ([127.0.0.1]:59289 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZKc-0002iN-TI for submit@debbugs.gnu.org; Tue, 23 Mar 2021 01:11:55 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:38467) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOZKa-0002i5-RD; Tue, 23 Mar 2021 01:11:53 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id E892813F4; Tue, 23 Mar 2021 01:11:46 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 01:11:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=wybaihPhH17Op27Ve rjCZANcJ6v7ziL1qmf92ho7/Ps=; b=pmHzZxAHMWUl71ZqS+t3OMHmTjpXYYB/5 OBvA0IC0vdBX7d7UovyCQ9k8lzWxH2hZplXrA0YljWIE/xYeCrWEZDWRj+697Q6Z JnjQLvtqDFTyfVxHDbb+95JU+rbiqW9zTp0tpx6Dv3AqmBF7ljILmRSEWVj+W34X Co2MTr72eglXrYSYE55r1R8H7qSQGUrqYtZd6uY7Ns+FIrwE1qM5wUUoh+fxt1Na 0n+SX1vO1sCLM6G0JBA5NvTpw5qBD6ibI8DORF0c1I0WezqNPf4dMZ+jjUWnNQSL s57ULcbA55ObOtkqvIUvWnaPwT5cBydswLlf16E89TIHWdtdW98ew== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgkedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffuffgjkfgfgggtsehttdertd dtredtnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpedtheehudeltefgffdtgffhhedvvd fhffelfeetteffhfekueeglefhhfdvteeuveenucfkphepjeegrdduudeirddukeeirdeg geenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrg hmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 1F221108005F; Tue, 23 Mar 2021 01:11:46 -0400 (EDT) From: Timothy Sample Date: Tue, 23 Mar 2021 01:11:45 -0400 In-Reply-To: bug's message of "Tue, 23 Mar 2021 04:53:56 +0000" Message-ID: <8735wmo3n2.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) reassign 47336 guix-patches thanks Oops! I sent this to the wrong list. My apologies. From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 23 Mar 2021 09:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample , 47336@debbugs.gnu.org, Mathieu Othacehe Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161649270212351 (code B ref 47336); Tue, 23 Mar 2021 09:46:01 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 09:45:02 +0000 Received: from localhost ([127.0.0.1]:59537 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOdav-0003Cz-Cr for submit@debbugs.gnu.org; Tue, 23 Mar 2021 05:45:01 -0400 Received: from mail-wm1-f50.google.com ([209.85.128.50]:55998) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOdat-0003Ck-Fq for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 05:44:59 -0400 Received: by mail-wm1-f50.google.com with SMTP id 12so10656916wmf.5 for <47336@debbugs.gnu.org>; Tue, 23 Mar 2021 02:44:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:in-reply-to:references:date:message-id:mime-version :content-transfer-encoding; bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=; b=a1kWhOty04rSkuZn+yOOy75QNSCxZU3tKaRoQzcBkwzUhLzQRGMidjhxwTCzmeQctS ujC55BPtDGxZVtFweOzRrWLjaea8ykqig+xmrybhktNCqYZZ9OwoS/Hx5fBtX7AKaHJs bSurhEv0RzAsjKnBE7drV8Hi2KxB7QCqXt0RfVdthLcy28j6WCVUSppEVi7ucmgMb2ph skLA9ocM+AYUU22B51RMDoyTuq4UwLQ7Ox9FMI2ApiZnNUunp+TRF+LWf6QxVKmXpgkB fUdTJFwHRzd6iuFDxCbUNBDAH2vITfyRpBbTFvGPJXcj1v3Vqm+xcmi2lixcnYGJmZsk nztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=xEawUxjs859Ktm1EBRae2ZC8JdLMCeoA9Tx4HO3D6Gw=; b=j1s02Fyb9xGS3D9uIpCmDT7+9dPrkxnTE2BtP3W6bowPwzQN5MrTntBrrV4Xgzyo7h Hqv2mSJyi7t5AggIjot1kn4HxsvvFjfrMu8SMA+pp8gKHtWTCyYcZo3g66mud9s/7OYm KPmRsdWM+Sh21aEL/tghxI8x9Z9NHPH15tqpHGFDxQTfIFVCm3C5TwifjapaC8z62hIW Rr+atHIUYVtrEa/TRjvF7eBB1BYIoEmQfqcrLEhRWCV3y6LLpHrgSItWqVd68JJcw5aN Ck9Pn3VetH+QPk9n3tEgrbq2/xkqNGMInzDZUW6DPQ/9QqirvJAK9aYKVnFbda0jvHXP nu7w== X-Gm-Message-State: AOAM531oRr/FTg6xo2XF6gFkoebVpwVtFhTSEAHbdnoCK9klky+yS3/9 JcIQxjFzAdChFx4cNLG6q7YAG9RZo2I= X-Google-Smtp-Source: ABdhPJzJrqraIVEvPjdWBzhyCgBvir/3yhgtfd13IxE/sysUfOFhuI40WjeXUCiXzi1VC5UM04lndw== X-Received: by 2002:a1c:66c4:: with SMTP id a187mr2474403wmc.164.1616492693386; Tue, 23 Mar 2021 02:44:53 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id s83sm2074886wmf.26.2021.03.23.02.44.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Mar 2021 02:44:53 -0700 (PDT) From: zimoun In-Reply-To: <87eeg6o50b.fsf@ngyro.com> References: <87eeg6o50b.fsf@ngyro.com> Date: Tue, 23 Mar 2021 10:35:53 +0100 Message-ID: <86sg4mnreu.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Timothy, (CC Mathieu to advice if it could be a feature of Cuirass.) On Tue, 23 Mar 2021 at 00:42, Timothy Sample wrote: > This patch series adds Disarchive assembly (backed by SWH lookup) as a > fallback for downloads. Awesome! > You also need to make sure that regular downloads are unavailable. I do > this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Cu= rl-fetch=E2=80=9D in > =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of U= RLs with =E2=80=98()=E2=80=99: > > (let try ((uri (append uri content-addressed-uris))) > (match '() ; uri > ...)) > > Now you can ask Guix for a recent .tar.gz source package: > > $ ./pre-inst-env guix build --no-substitutes -S python-httpretty Neat! Now, there is a way to easily check the coverage, right? Since SWH is ingesting the tarball using , there is now a mean to report what Guix is able to rebuild. > Checking httpretty-1.0.5 digest... ok What happens if it is not ok? > Assembling the tarball httpretty-1.0.5.tar > Checking httpretty-1.0.5.tar digest... ok > Assembling the Gzip file httpretty-1.0.5.tar.gz > Checking httpretty-1.0.5.tar.gz digest... ok > Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpret= ty-1.0.5.tar.gz Where is the assembly done? In /tmp/, right? > successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpre= tty-1.0.5.tar.gz.drv Just to be sure, when does Guix check the integrity checksum? I mean, does Guix check the checksum after =E2=80=99disassemble=E2=80=99 re-assembl= ed the source? > First, it looks up the metadata on my server. This is fine for a demo, > but not what we want forever. The patch series supports adding > several As we talked before, how does the database scale? Do you have some numbers for the current demo? In order to try to extrapolate what does it mean for a server to =C2=ABstore the metadata=C2=BB. > mirrors for looking up the metadata. In the past, we talked about > putting everything on one or a few of the big Git hosting platforms like > GitHub or Gitlab. That way, it would be easily picked up by SWH and > archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to = build the > metadata, and a little script that moves it from the build server to my > Web server. It would be simple enough to adjust that script to push it > to a remote Git repo. (Of course, the next step is to move this setup > to Guix infrastructure.) Thoughts? Maybe this database could be a package, say =E2=80=9Cguix-tarball-db=E2=80= =9D, updated in agreement with the package =E2=80=9Cguix=E2=80=9D. The source of this =E2=80=9Cguix-tarball-db=E2=80=9D would be a remote big Git hosting platfor= ms like GitHub or whatever and not stored on Guix infrastructure, or maybe stored on Guix infra. Regularly, i.e., when the package =E2=80=9Cguix=E2=80=9D is updated, in the= same time, the package =E2=80=9Cguix-tarball-db=E2=80=9D is updated too. The =E2=80= =9Cguix lint -c archival=E2=80=9D sends the saving request to SWH. Even if this saving req= uest should be automated soon. :-) Then if Cuirass would have a feature to disassemble and update the Git repo. Last, a service should run as your demo. But for long-term, this service could disappear=E2=80=93=E2=80=93assuming SWH not :-). Therefore, = we could imagine installing =E2=80=9Cguix-tarball-db=E2=80=9D then tweak some parame= ters of the guix-daemon and =E2=80=9Cguix build =E2=80=9D. Both installing and bu= ilding would fetch from SWH if both upstream disappear. Or this =E2=80=9Cguix-tarball-db=E2=80=9D should not be a plain package but= only an input as origin for the package =E2=80=9Cguix=E2=80=9D. > Hopefully everything else is more-or-less fine. :) Thanks! That=E2=80=99s awesome! Cheers, simon From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 23 Mar 2021 14:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: zimoun Cc: Mathieu Othacehe , 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161650988425671 (code B ref 47336); Tue, 23 Mar 2021 14:32:02 +0000 Received: (at 47336) by debbugs.gnu.org; 23 Mar 2021 14:31:24 +0000 Received: from localhost ([127.0.0.1]:60913 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOi44-0006fy-BI for submit@debbugs.gnu.org; Tue, 23 Mar 2021 10:31:24 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:56299) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOi42-0006fk-Ml for 47336@debbugs.gnu.org; Tue, 23 Mar 2021 10:31:23 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 842A05C00D7; Tue, 23 Mar 2021 10:31:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 10:31:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=MZOGcucNXJCcVOURYWb89EzsG/A48KOsBZjXvVtAo MM=; b=MQKGGVHnN1CXz9SGX+OTCpaG7PZ/x0DwKjIXqjWZcZ3Luo7adfatsuHAR j0mV3bvla//JvxvUlipL04UE5jFWH8DSnCOVqR9dBXJ9xCoL55XVT36h0lQwD8tT 9a+8VDrAnjzIkIsYohhK257/H8AFRdPXF8ggD2tpfvNY1QySvlFGH9EnYb33Wpr9 5g8Ner1rxahmOFLpTqbqUFQ61IxzXNloA/UyYJ8bUyXuoq9fTonPe1GzaIAh8jgY z8mOB8/v+dtQ11PURRpSCFWJOeBZee6nbuX/xCt8PRBF9dr19CfFaraYxG0y7iQo KrD7ZVIXJGMxyF5C4WgJJ+ymR0VNg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudegiedgieeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf grthhtvghrnhepledvlefhtdfhgfeuhfekiefhuefhfeevieeuheeuuedufefhudegffeg vedvjeefnecuffhomhgrihhnpehgnhhurdhorhhgpdhnghihrhhordgtohhmpdehnecukf hppeejgedrudduiedrudekiedrgeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepshgrmhhplhgvthesnhhghihrohdrtghomh X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 6D8F51080066; Tue, 23 Mar 2021 10:31:16 -0400 (EDT) From: Timothy Sample References: <87eeg6o50b.fsf@ngyro.com> <86sg4mnreu.fsf@gmail.com> Date: Tue, 23 Mar 2021 10:31:15 -0400 In-Reply-To: <86sg4mnreu.fsf@gmail.com> (zimoun's message of "Tue, 23 Mar 2021 10:35:53 +0100") Message-ID: <87sg4mt00c.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi zimoun, You make a lot of good points here. Let me at least provide some quick answers even if I=E2=80=99m not ready to comment on some of the bigger pict= ure stuff. zimoun writes: > (CC Mathieu to advice if it could be a feature of Cuirass.) So far I have been using Cuirass with only a tiny patch. I=E2=80=99m not s= ure we need anything more than what Cuirass already provides. (The tiny patch is for allowing sorting the =E2=80=9Clatestbuilds=E2=80=9D results by= =E2=80=9Cstoptime=E2=80=9D and =E2=80=9Cid=E2=80=9D. This in turn allows paging through all the build= s from the API.) > On Tue, 23 Mar 2021 at 00:42, Timothy Sample wrote: > >> Now you can ask Guix for a recent .tar.gz source package: >> >> $ ./pre-inst-env guix build --no-substitutes -S python-httpretty > > Neat! Now, there is a way to easily check the coverage, right? Since > SWH is ingesting the tarball using , > there is now a mean to report what Guix is able to rebuild. I=E2=80=99m not sure I fully understand. Disarchive covers about 4,300 Gzi= p=E2=80=99ed tarballs (no XZ yet). There are about 100 for which compression parameters cannot be found, and a handful (about 5) that have a particularly funny idea about what a tarball is. The metadata builds for my database started one week ago and have been continuously updating since then. Are you asking if we could check what SWH has? Yes! Each metadata file contains the SWHID of the input directory. You could use Disarchive to get this value or a simple =E2=80=9Cgrep swhid=E2=80=9D would= do it. :) $ curl https://disarchive.ngyro.com/sha256/67989614004773db349791c37675= efb914d084bdb221356a05e4369c35e7eb62 | grep swhid It would be neat to have a big database of archive coverage from Guix 1.0 through to the present. It=E2=80=99s quite a big project though. Of course, you know all about the SWH rate limit.... >> Checking httpretty-1.0.5 digest... ok > > What happens if it is not ok? For that particular digest, it means the source directory is wrong. Since we get the source from SWH, it means that the SWH archive is wrong. You will have to look elsewhere, I guess (this seems pretty unlikely). (There is a vanishing possibility that Disarchive miscomputed the SWHID and managed to come up with a different, but still valid SWHID....) The other digest checks are more likely to fail. They would indicate that Disarchive no longer knows how to interpret the metadata. Maybe there will be a subtle bug in Disarchive 0.3.0 that causes this. Either use an old version of Disarchive or try to fix the current version. :) I worry about this, because it would be annoying, but the metadata does have all the information needed to recover the original archive, so nothing is really lost (except the user=E2=80=99s time). >> Assembling the tarball httpretty-1.0.5.tar >> Checking httpretty-1.0.5.tar digest... ok >> Assembling the Gzip file httpretty-1.0.5.tar.gz >> Checking httpretty-1.0.5.tar.gz digest... ok >> Copying result to >> /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz > > Where is the assembly done? In /tmp/, right? Yes. >> successfully built >> /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpretty-1.0.5.tar.gz.drv > > Just to be sure, when does Guix check the integrity checksum? I mean, > does Guix check the checksum after =E2=80=99disassemble=E2=80=99 re-assem= bled the source? Disarchive checks the result against the metadata to make sure it didn=E2= =80=99t make a mistake. Guix also checks the final result to make sure the fixed-output derivation is correct. A fixed-output derivation is basically just a checksum with a hint about how the data can be obtained. Guix really only cares about the checksum, the hint can do whatever as long as it produces the result Guix wants. With this patch series, Disarchive is part of the hint. >> First, it looks up the metadata on my server. This is fine for a demo, >> but not what we want forever. The patch series supports adding >> several > > As we talked before, how does the database scale? Do you have some > numbers for the current demo? In order to try to extrapolate what does > it mean for a server to =C2=ABstore the metadata=C2=BB. With =E2=80=9Cgzip -9=E2=80=9D, the average metadata file is 6.8KiB. It=E2= =80=99s pretty manageable. There=E2=80=99s room for improvement on the Disarchive side, t= oo. It still stores some redundant information. Uncompressed, it=E2=80=99s more like 112KiB per file. This is still pretty okay, really. It means we might hit tens of GiB over a couple years. (It would take just over 100GiB to store a million uncompressed metadata files.) The compression ratio is what drove me to skip Git for now. >> mirrors for looking up the metadata. In the past, we talked about >> putting everything on one or a few of the big Git hosting platforms like >> GitHub or Gitlab. That way, it would be easily picked up by SWH and >> archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to= build the >> metadata, and a little script that moves it from the build server to my >> Web server. It would be simple enough to adjust that script to push it >> to a remote Git repo. (Of course, the next step is to move this setup >> to Guix infrastructure.) Thoughts? > > Maybe this database could be a package, say =E2=80=9Cguix-tarball-db=E2= =80=9D, updated > in agreement with the package =E2=80=9Cguix=E2=80=9D. The source of this > =E2=80=9Cguix-tarball-db=E2=80=9D would be a remote big Git hosting platf= orms like > GitHub or whatever and not stored on Guix infrastructure, or maybe > stored on Guix infra. > > Regularly, i.e., when the package =E2=80=9Cguix=E2=80=9D is updated, in t= he same time, > the package =E2=80=9Cguix-tarball-db=E2=80=9D is updated too. The =E2=80= =9Cguix lint -c > archival=E2=80=9D sends the saving request to SWH. Even if this saving r= equest > should be automated soon. :-) > > Then if Cuirass would have a feature to disassemble and update the Git > repo. > > Last, a service should run as your demo. But for long-term, this > service could disappear=E2=80=93=E2=80=93assuming SWH not :-). Therefore= , we could > imagine installing =E2=80=9Cguix-tarball-db=E2=80=9D then tweak some para= meters of the > guix-daemon and =E2=80=9Cguix build =E2=80=9D. Both installing and = building would > fetch from SWH if both upstream disappear. > > Or this =E2=80=9Cguix-tarball-db=E2=80=9D should not be a plain package b= ut only an > input as origin for the package =E2=80=9Cguix=E2=80=9D. This is an interesting idea, but one that I would have to think about more. :) -- Tim From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 27 Mar 2021 10:40:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: Mathieu Othacehe , 47336@debbugs.gnu.org, zimoun Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.16168415729325 (code B ref 47336); Sat, 27 Mar 2021 10:40:01 +0000 Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:39:32 +0000 Received: from localhost ([127.0.0.1]:43457 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6Ls-0002QL-Ay for submit@debbugs.gnu.org; Sat, 27 Mar 2021 06:39:32 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33608) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6Lp-0002Q7-Hf for 47336@debbugs.gnu.org; Sat, 27 Mar 2021 06:39:30 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34494) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lQ6Lj-00046H-Lv; Sat, 27 Mar 2021 06:39:23 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41636 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lQ6Lh-0003Y1-1l; Sat, 27 Mar 2021 06:39:22 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> <86sg4mnreu.fsf@gmail.com> <87sg4mt00c.fsf@ngyro.com> Date: Sat, 27 Mar 2021 11:39:18 +0100 In-Reply-To: <87sg4mt00c.fsf@ngyro.com> (Timothy Sample's message of "Tue, 23 Mar 2021 10:31:15 -0400") Message-ID: <87eeg0284p.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi! Timothy Sample skribis: > With =E2=80=9Cgzip -9=E2=80=9D, the average metadata file is 6.8KiB. It= =E2=80=99s pretty > manageable. There=E2=80=99s room for improvement on the Disarchive side,= too. > It still stores some redundant information. Uncompressed, it=E2=80=99s m= ore > like 112KiB per file. This is still pretty okay, really. It means we > might hit tens of GiB over a couple years. (It would take just over > 100GiB to store a million uncompressed metadata files.) The compression > ratio is what drove me to skip Git for now. If needed, the sexp serialization could still be made more compact: using =E2=80=98write=E2=80=99 instead of =E2=80=98pretty-print=E2=80=99, sh= ortening field names (but that=E2=80=99d be incompatible). We could also use CBOR or canonical sexp serialization, though maybe gzipped sexps are more compact than what we could achieve? Anyway, these are surface syntax optimizations that can always be made at a later point in time when we feel a need for them. Ludo=E2=80=99. From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 27 Mar 2021 10:41:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.16168416259432 (code B ref 47336); Sat, 27 Mar 2021 10:41:01 +0000 Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:40:25 +0000 Received: from localhost ([127.0.0.1]:43461 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6Mj-0002S3-K7 for submit@debbugs.gnu.org; Sat, 27 Mar 2021 06:40:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33754) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6Mh-0002Rp-A1 for 47336@debbugs.gnu.org; Sat, 27 Mar 2021 06:40:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34499) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lQ6Mc-0004bG-37; Sat, 27 Mar 2021 06:40:18 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41638 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lQ6Ma-0006GK-4Q; Sat, 27 Mar 2021 06:40:17 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> Date: Sat, 27 Mar 2021 11:40:14 +0100 In-Reply-To: <20210323045213.9419-1-samplet@ngyro.com> (Timothy Sample's message of "Tue, 23 Mar 2021 00:52:12 -0400") Message-ID: <87a6qo2835.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Timothy Sample skribis: > * guix/swh.scm (swh-directory-download): New procedure (with > implementation extracted from 'swh-download'). > (swh-download): Use it to download the revision directory. LGTM! From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 27 Mar 2021 10:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161684264210939 (code B ref 47336); Sat, 27 Mar 2021 10:58:02 +0000 Received: (at 47336) by debbugs.gnu.org; 27 Mar 2021 10:57:22 +0000 Received: from localhost ([127.0.0.1]:43471 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6d8-0002qN-Ey for submit@debbugs.gnu.org; Sat, 27 Mar 2021 06:57:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36022) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lQ6d6-0002q9-Om for 47336@debbugs.gnu.org; Sat, 27 Mar 2021 06:57:21 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34573) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lQ6d1-0006gL-8A; Sat, 27 Mar 2021 06:57:15 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41646 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lQ6d0-00007a-Mw; Sat, 27 Mar 2021 06:57:15 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20210323045213.9419-1-samplet@ngyro.com> <20210323045213.9419-2-samplet@ngyro.com> Date: Sat, 27 Mar 2021 11:57:12 +0100 In-Reply-To: <20210323045213.9419-2-samplet@ngyro.com> (Timothy Sample's message of "Tue, 23 Mar 2021 00:52:13 -0400") Message-ID: <87y2e8zwxj.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi! Timothy Sample skribis: > * guix/download.scm (%disarchive-mirrors): New variable. > (%disarchive-mirror-file): New variable. > (built-in-download): Add 'disarchive-mirrors' keyword argument and > pass its value along to the 'builtin:download' derivation. > (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'. > * guix/scripts/perform-download.scm (perform-download): Read > Disarchive mirrors from the environment and pass them to > 'url-fetch'. > * guix/build/download.scm (disarchive-fetch/any): New procedure. > (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to > make a list of URIs, and use the new procedure to fetch the file if > all other methods fail. [...] > + #:use-module (guix base16) > #:use-module (guix base64) > #:use-module (guix ftp-client) > #:use-module (guix build utils) > #:use-module (guix progress) > + #:use-module (guix swh) Maybe #:autoload them. > +(define* (disarchive-fetch/any uris file > + #:key (timeout 10)) > + "Fetch a Disarchive specification from any of URIS, assemble it, > +and write the output to FILE." > + (define (fetch-specification uris) > + (any (lambda (uri) > + (false-if-exception* > + (let-values (((port size) (http-fetch uri > + #:verify-certificate? = #t > + #:timeout timeout))) Perhaps add #:key (verify-certificate? #t) and have the caller pass it? Currently (guix scripts perform-download) sets it to #f, which is a good idea IMO. > + (match (and=3D> (resolve-module '(disarchive) #:ensure #f) > + (lambda (disarchive) > + (cons (module-ref disarchive '%disarchive-log-port) > + (module-ref disarchive 'disarchive-assemble)))) > + (#f #f) > + ((%disarchive-log-port . disarchive-assemble) > + (format #t "Trying to use Disarchive to assemble ~a~%" file) > + (match (fetch-specification uris) > + (#f #f) > + (spec (parameterize ((%disarchive-log-port (current-output-port))) > + (disarchive-assemble spec file #:resolver resolve))))))) So we would normally arrange so that the =E2=80=98guix=E2=80=99 package dep= ends on Disarchive, such that the above =E2=80=98resolve-module=E2=80=99 call works= when done via =E2=80=98guix perform-download=E2=80=99, right? In the #f case, perhaps we should print something like =E2=80=9CDisarchive = not found, bailing out=E2=80=9D? That=E2=80=99s all I have to say; it looks great to me! That=E2=80=99s quite a milestone, it=E2=80=99d be great to have that in the= upcoming release. Next we can discuss how to populate the Disarchive database and where to do that (or your hosting fees could easily skyrocket :-)). I suppose we could run that in Berlin and/or we could make an argument about using SWH or Inria resources for that. Thanks, Ludo=E2=80=99. From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 10 Apr 2021 20:53:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161808796210630 (code B ref 47336); Sat, 10 Apr 2021 20:53:02 +0000 Received: (at 47336) by debbugs.gnu.org; 10 Apr 2021 20:52:42 +0000 Received: from localhost ([127.0.0.1]:53418 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lVKaw-0002lO-1b for submit@debbugs.gnu.org; Sat, 10 Apr 2021 16:52:42 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47430) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lVKau-0002l9-OE for 47336@debbugs.gnu.org; Sat, 10 Apr 2021 16:52:41 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50810) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lVKap-00022A-DR; Sat, 10 Apr 2021 16:52:35 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=42156 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lVKao-000323-Sf; Sat, 10 Apr 2021 16:52:35 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> <87a6qo2835.fsf_-_@gnu.org> Date: Sat, 10 Apr 2021 22:52:33 +0200 In-Reply-To: <87a6qo2835.fsf_-_@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Sat, 27 Mar 2021 11:40:14 +0100") Message-ID: <87blal2772.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Ping! :-) Ludovic Court=C3=A8s skribis: > Timothy Sample skribis: > >> * guix/swh.scm (swh-directory-download): New procedure (with >> implementation extracted from 'swh-download'). >> (swh-download): Use it to download the revision directory. > > LGTM! From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 26 Apr 2021 09:51:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161943061512602 (code B ref 47336); Mon, 26 Apr 2021 09:51:02 +0000 Received: (at 47336) by debbugs.gnu.org; 26 Apr 2021 09:50:15 +0000 Received: from localhost ([127.0.0.1]:44619 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laxsd-0003HC-7V for submit@debbugs.gnu.org; Mon, 26 Apr 2021 05:50:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36402) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laxsa-0003Gx-TV for 47336@debbugs.gnu.org; Mon, 26 Apr 2021 05:50:14 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52159) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1laxsV-0002Az-Gf; Mon, 26 Apr 2021 05:50:07 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55298 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1laxsL-0005MF-CU; Mon, 26 Apr 2021 05:50:06 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> <87a6qo2835.fsf_-_@gnu.org> Date: Mon, 26 Apr 2021 11:49:55 +0200 In-Reply-To: <87a6qo2835.fsf_-_@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Sat, 27 Mar 2021 11:40:14 +0100") Message-ID: <87v989e5oc.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Timothy, Ping=C2=B2! Let me know if you=E2=80=99re like me to apply the patches on your behalf. Ludo=E2=80=99. Ludovic Court=C3=A8s skribis: > Timothy Sample skribis: > >> * guix/swh.scm (swh-directory-download): New procedure (with >> implementation extracted from 'swh-download'). >> (swh-download): Use it to download the revision directory. > > LGTM! From unknown Sat Aug 09 13:19:09 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Timothy Sample Subject: bug#47336: closed (Re: bug#47336: Disarchive as a fallback for downloads) Message-ID: References: <87o8dzkunk.fsf_-_@ngyro.com> <87eeg6o50b.fsf@ngyro.com> X-Gnu-PR-Message: they-closed 47336 X-Gnu-PR-Package: guix-patches Reply-To: 47336@debbugs.gnu.org Date: Wed, 28 Apr 2021 02:31:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1619577062-28565-1" This is a multi-part message in MIME format... ------------=_1619577062-28565-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #47336: Disarchive as a fallback for downloads which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 47336@debbugs.gnu.org. --=20 47336: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47336 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1619577062-28565-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 47336-done) by debbugs.gnu.org; 28 Apr 2021 02:30:51 +0000 Received: from localhost ([127.0.0.1]:51379 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lbZyU-0007QK-Ou for submit@debbugs.gnu.org; Tue, 27 Apr 2021 22:30:50 -0400 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]:46999) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lbZyR-0007Pz-0T for 47336-done@debbugs.gnu.org; Tue, 27 Apr 2021 22:30:48 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 6D226EA3; Tue, 27 Apr 2021 22:30:41 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 27 Apr 2021 22:30:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=4XqfWTcO+597/Uu3CUK5wZtE/MGAF4i4wRDNBVMPa PE=; b=dnUGU82lUM+ZvPwwEaP024vphaFUxxnh2jZ0RakQDnp5zCitB4Ze3qUBm 6FnYQbD6G3F0WiTBY5HdhCfTZTP59mJKEaKkMnHrowEAFUO7ZnFC171Y+Zxypmxw pY7gCYMA2h8y3lPh/oQNqwYjJnw/DHS6/g/uUbc+rvKld6F0fstpyD+iqAD4A6Dh g0KKAstsgtiACMuKqCh+iylNoRabaPhW+nPtXH47IovAArQuirOcL0IVVBz371S+ E+f1SplzgwrurRxuo8BeMLHDvLgb48RvIyCt442JytOAjR74SwzdY7JxG0h7a4ZF XjBzmPCr3YxZn3X3YGXgbAoRxE73g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvddvuddgheelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf grthhtvghrnhepvdelvefhheegudevjedvfedtveeileffveehveejgefgteetteejteel tdefueevnecuffhomhgrihhnpehgnhhurdhorhhgnecukfhppeejgedrudduiedrudekie drgeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep shgrmhhplhgvthesnhhghihrohdrtghomh X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 27 Apr 2021 22:30:40 -0400 (EDT) From: Timothy Sample To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#47336: Disarchive as a fallback for downloads References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> <87a6qo2835.fsf_-_@gnu.org> <87v989e5oc.fsf_-_@gnu.org> Date: Tue, 27 Apr 2021 22:30:39 -0400 In-Reply-To: <87v989e5oc.fsf_-_@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s?= =?utf-8?Q?=22's?= message of "Mon, 26 Apr 2021 11:49:55 +0200") Message-ID: <87o8dzkunk.fsf_-_@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47336-done Cc: 47336-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi, Ludovic Court=C3=A8s writes: > Ping=C2=B2! > > Let me know if you=E2=80=99re like me to apply the patches on your behalf. No, no. I=E2=80=99m just a little distracted over here. I just pushed this series with the updates you suggested (using #:autoload, passing #:verify-certificates?, and being a bit more chatty). Sorry for the delay and thanks for the reminder. Next, I=E2=80=99ll convert my Cuirass 0.x setup to a Cuirass 1.x setup, and= then I can start a discussion about moving the metadata builds to ci.guix.gnu.org. Also, to answer your other question: > So we would normally arrange so that the =E2=80=98guix=E2=80=99 package d= epends on > Disarchive, such that the above =E2=80=98resolve-module=E2=80=99 call wor= ks when done > via =E2=80=98guix perform-download=E2=80=99, right? That=E2=80=99s the idea. I=E2=80=99m not confident about updating the =E2= =80=98guix=E2=80=99 package myself, though.... -- Tim ------------=_1619577062-28565-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000 Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs3-0001pw-WB for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:51240) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs2-0001pp-0u for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59600) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYs1-0007wV-QR for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYry-0007rZ-7A for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id D60141373 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/ YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) From: Timothy Sample To: bug-guix@gnu.org Subject: Disarchive as a fallback for downloads Date: Tue, 23 Mar 2021 00:42:12 -0400 Message-ID: <87eeg6o50b.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@ngyro.com; helo=wout3-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) Hello, This patch series adds Disarchive assembly (backed by SWH lookup) as a fallback for downloads. To try it, make sure you are running the daemon in an environment with Disarchive available: $ ./pre-inst-env guix environment --ad-hoc guile disarchive # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild Don=E2=80=99t forget to stop your existing Guix Daemon. :) You also need to make sure that regular downloads are unavailable. I do this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl= -fetch=E2=80=9D in =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of URL= s with =E2=80=98()=E2=80=99: (let try ((uri (append uri content-addressed-uris))) (match '() ; uri ...)) Now you can ask Guix for a recent .tar.gz source package: $ ./pre-inst-env guix build --no-substitutes -S python-httpretty You should see: Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v= diwlv19y9-httpretty-1.0.5.tar.gz Assembling the directory httpretty-1.0.5 Downloading from Software Heritage... 7903d608efc89c14afb4d692a3721156e31a43e2/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING [...] Checking httpretty-1.0.5 digest... ok Assembling the tarball httpretty-1.0.5.tar Checking httpretty-1.0.5.tar digest... ok Assembling the Gzip file httpretty-1.0.5.tar.gz Checking httpretty-1.0.5.tar.gz digest... ok Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty= -1.0.5.tar.gz successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett= y-1.0.5.tar.gz.drv There=E2=80=99s lots to talk about though.... First, it looks up the metadata on my server. This is fine for a demo, but not what we want forever. The patch series supports adding several mirrors for looking up the metadata. In the past, we talked about putting everything on one or a few of the big Git hosting platforms like GitHub or Gitlab. That way, it would be easily picked up by SWH and archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to bu= ild the metadata, and a little script that moves it from the build server to my Web server. It would be simple enough to adjust that script to push it to a remote Git repo. (Of course, the next step is to move this setup to Guix infrastructure.) Thoughts? On the code level, there were two things I couldn=E2=80=99t figure out for myself. I made the mirror list just simple strings. AIUI, the client and the daemon have to agree about the format of the mirror list. Given that running old daemons is common, changing the format is difficult. Is it worth it to copy the more flexible interface used by the content addressed mirrors? If yes, do I have to do the same =E2=80=98module-autolo= ad!=E2=80=99 dance to use =E2=80=98bytevector->base16-string=E2=80=99? :) (I probably = would have just copied it, but that part confused me a bit.) I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel= l, just =E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D). It feels weird to use= a bunch of host-side modules from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module. This = is okay because =E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. = It=E2=80=99s more like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2= =80=9D, which is why importing host-side modules is OK... right? Hopefully everything else is more-or-less fine. :) -- Tim ------------=_1619577062-28565-1-- From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Timothy Sample Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 28 Apr 2021 07:03:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: control@debbugs.gnu.org, 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161959332523960 (code B ref 47336); Wed, 28 Apr 2021 07:03:02 +0000 Received: (at 47336) by debbugs.gnu.org; 28 Apr 2021 07:02:05 +0000 Received: from localhost ([127.0.0.1]:51583 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lbeCz-0006EO-4R for submit@debbugs.gnu.org; Wed, 28 Apr 2021 03:02:05 -0400 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:55273) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lbeCw-0006Dp-3A; Wed, 28 Apr 2021 03:02:03 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id B99DC5C00BD; Wed, 28 Apr 2021 03:01:56 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Wed, 28 Apr 2021 03:01:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=73wGy0hp+Ms9wxJ0OS/8kZyUHxq2hBRI86EwfbO2J r0=; b=HElWpid1QUvLXjr5widnlf9WrYsJxxlpXpeyv320R5gGrr+hO2jwvJzQH YJCUsKt1tmiwdnjHRyr3c6TuW2p2Zd8fiQLeu35Dvz+nHfF3sztWCrY/Rp8tIM1W 0rH46o8C0+da6KXnID89d2pV3ptgSam4Yrvmk2dZ92xX7Dd+5d7UOL2Aq9Y8X+Y8 31xIqXrPR/IH6qUCC84azZuzUYQowvveYzB8f5TqlmeiX2RoSNiQTBhnXF9dkA2W wlaMyu24IJk5St/pcyY2XXpnvLCP6FoIqz8KLAOuuLw8y4vIeZoTEpn9HVRZw8OX 2d9E1YhUbj9u/TSaq6IzDs7dK0g8Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvddvuddguddufecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufhffjgfkfgggtgfgsehtqhertddtreejnecuhfhrohhmpefvihhm ohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehnghihrhhordgtohhmqeenucggtf frrghtthgvrhhnpeeiudeuhfeggeelleevheegudfguefhieekffdtveeilefglefhvddt gfeiheetgfenucfkphepjeegrdduudeirddukeeirdeggeenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgt ohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 28 Apr 2021 03:01:56 -0400 (EDT) From: Timothy Sample References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> <87a6qo2835.fsf_-_@gnu.org> <87v989e5oc.fsf_-_@gnu.org> <87o8dzkunk.fsf_-_@ngyro.com> Date: Wed, 28 Apr 2021 03:01:55 -0400 In-Reply-To: <87o8dzkunk.fsf_-_@ngyro.com> (Timothy Sample's message of "Tue, 27 Apr 2021 22:30:39 -0400") Message-ID: <87h7jqlwnw.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) reopen 47336 thanks Hi again, Timothy Sample writes: > I just pushed this series [...] And broke =E2=80=9Cguix pull=E2=80=9D!! (I somehow fooled myself into thin= king that I had already tested with =E2=80=9Cguix pull --url=3D...=E2=80=9D locally.) = I reverted the offending commit. It turns out that adding a reference from =E2=80=9C(guix build download)=E2= =80=9D to =E2=80=9C(guix swh)=E2=80=9D breaks =E2=80=9Ccompute-guix-derivation=E2=80= =9D in =E2=80=9Cbuild-aux/build-self.scm=E2=80=9D. This is because =E2=80=9C(guix= swh)=E2=80=9D references =E2=80=9C(json)=E2=80=9D, which is not available in the =E2=80=9Ccompute-gu= ix-derivation=E2=80=9D environment. I tried mimicking the =E2=80=9Cfake-git=E2=80=9D trick, but i= t didn=E2=80=99t work (I guess it needs the =E2=80=9Cdefine-json-mapping=E2=80=9D macro at compil= e time). Everything works if I remove the #:autoload for =E2=80=9C(guix swh)=E2=80= =9D and put ;; If we import (guix swh) directly, we introduce a compile-time ;; dependency on Guile-JSON. This breaks the "build-self" code, which ;; needs to build this module without Guile-JSON. Hence, we track ;; down the following procedure at runtime. (define swh-download-directory (module-ref (resolve-module '(guix swh)) 'swh-download-directory)) inside of =E2=80=9Cdisarchive-fetch/any=E2=80=9D (just before it=E2=80=99s = needed). Does this approach look okay? -- Tim From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Apr 2021 07:50:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.161968254815808 (code B ref 47336); Thu, 29 Apr 2021 07:50:01 +0000 Received: (at 47336) by debbugs.gnu.org; 29 Apr 2021 07:49:08 +0000 Received: from localhost ([127.0.0.1]:54752 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lc1Q4-00046u-0A for submit@debbugs.gnu.org; Thu, 29 Apr 2021 03:49:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33872) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lc1Q2-00046e-VE for 47336@debbugs.gnu.org; Thu, 29 Apr 2021 03:49:07 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37902) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lc1Px-0008Ar-M5; Thu, 29 Apr 2021 03:49:01 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=60830 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lc1Pw-0006jK-HO; Thu, 29 Apr 2021 03:49:00 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> <87a6qo2835.fsf_-_@gnu.org> <87v989e5oc.fsf_-_@gnu.org> <87o8dzkunk.fsf_-_@ngyro.com> <87h7jqlwnw.fsf@ngyro.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 10 =?UTF-8?Q?Flor=C3=A9al?= an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 29 Apr 2021 09:48:59 +0200 In-Reply-To: <87h7jqlwnw.fsf@ngyro.com> (Timothy Sample's message of "Wed, 28 Apr 2021 03:01:55 -0400") Message-ID: <874kfpo7is.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! Timothy Sample skribis: > And broke =E2=80=9Cguix pull=E2=80=9D!! (I somehow fooled myself into th= inking that I > had already tested with =E2=80=9Cguix pull --url=3D...=E2=80=9D locally.)= I reverted the > offending commit. You can test with =E2=80=98guix pull=E2=80=99 (you need to make sure to spe= cify the right file:// URL *and* branch), or you can run =E2=80=9Cmake as-derivation= =E2=80=9D. > It turns out that adding a reference from =E2=80=9C(guix build download)= =E2=80=9D to > =E2=80=9C(guix swh)=E2=80=9D breaks =E2=80=9Ccompute-guix-derivation=E2= =80=9D in > =E2=80=9Cbuild-aux/build-self.scm=E2=80=9D. This is because =E2=80=9C(gu= ix swh)=E2=80=9D references > =E2=80=9C(json)=E2=80=9D, which is not available in the =E2=80=9Ccompute-= guix-derivation=E2=80=9D > environment. I tried mimicking the =E2=80=9Cfake-git=E2=80=9D trick, but= it didn=E2=80=99t work > (I guess it needs the =E2=80=9Cdefine-json-mapping=E2=80=9D macro at comp= ile time). > > Everything works if I remove the #:autoload for =E2=80=9C(guix swh)=E2=80= =9D and put > > ;; If we import (guix swh) directly, we introduce a compile-time > ;; dependency on Guile-JSON. This breaks the "build-self" code, which > ;; needs to build this module without Guile-JSON. Hence, we track > ;; down the following procedure at runtime. > (define swh-download-directory > (module-ref (resolve-module '(guix swh)) 'swh-download-directory)) > > inside of =E2=80=9Cdisarchive-fetch/any=E2=80=9D (just before it=E2=80=99= s needed). Does this > approach look okay? That=E2=80=99s one possibility. The patch below takes another approach. I think it aesthetically slightly more pleasant because we don=E2=80=99t have to play =E2=80=98resol= ve-module=E2=80=99 tricks for obscure reasons. WDYT? (It also fixes a format string argument mismatch.) Thanks! Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 853a2f328f..f100ff4aae 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -250,6 +250,7 @@ interface (FFI) of Guile.") (match-lambda (('guix 'config) #f) (('guix 'channels) #f) + (('guix 'build 'download) #f) ;autoloaded by (guix download) (('guix _ ...) #t) (('gnu _ ...) #t) (_ #f))) diff --git a/guix/build/download.scm b/guix/build/download.scm index 5431d7c682..ce31038b05 100644 --- a/guix/build/download.scm +++ b/guix/build/download.scm @@ -650,7 +650,7 @@ and write the output to FILE." (('swhid swhid) (match (string-split swhid #\:) (("swh" "1" "dir" id) - (format #t "Downloading from Software Heritage...~%" file) + (format #t "Downloading ~a from Software Heritage...~%" file) (false-if-exception* (swh-download-directory id output))) (_ #f))) diff --git a/guix/self.scm b/guix/self.scm index 3154d180ac..7181205610 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -878,7 +878,8 @@ itself." ("guix/store/schema.sql" ,(local-file "../guix/store/schema.sql"))) - #:extensions (list guile-gcrypt) + #:extensions (list guile-gcrypt + guile-json) ;for (guix swh) #:guile-for-build guile-for-build)) (define *extra-modules* --=-=-=-- From unknown Sat Aug 09 13:19:09 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Timothy Sample Subject: bug#47336: closed (Re: bug#47336: Disarchive as a fallback for downloads) Message-ID: References: <878s51knra.fsf_-_@ngyro.com> <87eeg6o50b.fsf@ngyro.com> X-Gnu-PR-Message: they-closed 47336 X-Gnu-PR-Package: guix-patches Reply-To: 47336@debbugs.gnu.org Date: Thu, 29 Apr 2021 17:25:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1619717102-24882-1" This is a multi-part message in MIME format... ------------=_1619717102-24882-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #47336: Disarchive as a fallback for downloads which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 47336@debbugs.gnu.org. --=20 47336: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D47336 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1619717102-24882-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 47336-done) by debbugs.gnu.org; 29 Apr 2021 17:24:21 +0000 Received: from localhost ([127.0.0.1]:56746 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lcAOj-0006SL-9o for submit@debbugs.gnu.org; Thu, 29 Apr 2021 13:24:21 -0400 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:59945) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lcAOd-0006S4-QT for 47336-done@debbugs.gnu.org; Thu, 29 Apr 2021 13:24:19 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 8D5655C00BE; Thu, 29 Apr 2021 13:24:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Thu, 29 Apr 2021 13:24:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=ewkAfvBZTTrYmtqsnnrWJdUthPPj67lMNOuqm4DOa 0k=; b=U1uuE/N1XVt/a3kHWQHkIlnoar6zOnGlnin6e6HE13tAk/pSPZh1GpNsv rXQvgNAD/KQgK4yBM23fF5ugJF0U3+SjnptmA3Y9jkGaJ7zcgGodAeuqOnxv7xME AAtzQzbm7GL5htnAhf/pkCcDidqGOQ06ED+9rCC+MWOeJ0IwGg4cWYbWuvP+Dk9n k/CwCc78wRThvQ+tTdxu5m4InLdwgl1Nd1dFzE+bOUfGFLTgA5ZeOjTJZW8Vhpv+ tojSFqcX29l6L6WMdArcB3CuHc0KmBGf1KZDB+cHGURV9IeWnekSvQz+fYikLmDB An58vsXDUyD4CKgQ3lkuVXgsuO0dg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvddvgedguddugecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufhffjgfkfgggtgfgsehtqhertddtreejnecuhfhrohhmpefvihhm ohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehnghihrhhordgtohhmqeenucggtf frrghtthgvrhhnpeeiudeuhfeggeelleevheegudfguefhieekffdtveeilefglefhvddt gfeiheetgfenucfkphepjeegrdduudeirddukeeirdeggeenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgt ohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 29 Apr 2021 13:24:10 -0400 (EDT) From: Timothy Sample To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#47336: Disarchive as a fallback for downloads References: <87eeg6o50b.fsf@ngyro.com> <20210323045213.9419-1-samplet@ngyro.com> <87a6qo2835.fsf_-_@gnu.org> <87v989e5oc.fsf_-_@gnu.org> <87o8dzkunk.fsf_-_@ngyro.com> <87h7jqlwnw.fsf@ngyro.com> <874kfpo7is.fsf@gnu.org> Date: Thu, 29 Apr 2021 13:24:09 -0400 In-Reply-To: <874kfpo7is.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Thu, 29 Apr 2021 09:48:59 +0200") Message-ID: <878s51knra.fsf_-_@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47336-done Cc: 47336-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hello, Ludovic Court=C3=A8s writes: > Timothy Sample skribis: > >> And broke =E2=80=9Cguix pull=E2=80=9D!! (I somehow fooled myself into t= hinking that I >> had already tested with =E2=80=9Cguix pull --url=3D...=E2=80=9D locally.= ) I reverted the >> offending commit. > > You can test with =E2=80=98guix pull=E2=80=99 (you need to make sure to s= pecify the > right file:// URL *and* branch), or you can run =E2=80=9Cmake as-derivati= on=E2=80=9D. I will definitely be more careful with this in the future. >> [...] Does this approach look okay? > > That=E2=80=99s one possibility. > > The patch below takes another approach. I think it aesthetically > slightly more pleasant because we don=E2=80=99t have to play =E2=80=98res= olve-module=E2=80=99 > tricks for obscure reasons. WDYT? This is exactly what I was hoping for, but I couldn=E2=80=99t quite connect= all the dots in =E2=80=9Cbuild-self.scm=E2=80=9D. Thanks! > (It also fixes a format string argument mismatch.) Good catch! I=E2=80=99ve pushed the updated patch and am closing the issue. :) -- Tim ------------=_1619717102-24882-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 23 Mar 2021 04:42:24 +0000 Received: from localhost ([127.0.0.1]:59241 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs3-0001pw-WB for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:24 -0400 Received: from lists.gnu.org ([209.51.188.17]:51240) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lOYs2-0001pp-0u for submit@debbugs.gnu.org; Tue, 23 Mar 2021 00:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59600) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYs1-0007wV-QR for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]:60517) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lOYry-0007rZ-7A for bug-guix@gnu.org; Tue, 23 Mar 2021 00:42:21 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id D60141373 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 23 Mar 2021 00:42:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=lFT6AC aQH3IIu01q7x1IIPLbiIT11PkFDblkw7NP2/w=; b=X/5rSkWH4GZsZKZYzp3VZ6 RK81gLWKGdsMoHzOd45oM8YzyLn6eVEE4/OYqMl2pU6v8VUcGrIu1j/joeBaXuN0 Bb/KFhU1sKX4wuHpIs5Zh5+xCegt15YHgjROoYKJ91UgaBSxWW2D1WOqRuSCi5h5 UMg02e9opVGo5/sDvbY0+CzQ5HFhM+T43vuhxZhRxN75lplEhwtYhdGqcRnJ4+X/ YxY9l4O0bujCtwPcwxCK8p/yCQbxF8roA1h28+EL5sCafANbdD/LRnvOaTV9YnKl 96BCxIAQnvElsdWFMYn+FoBFeuSLtxM7l7+dTqxAeZWbfikTmRfvRJ5a2Jrb39YA == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrudeghedgjeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffufffkfgggtgfgsehtqhertd dtreejnecuhfhrohhmpefvihhmohhthhihucfurghmphhlvgcuoehsrghmphhlvghtsehn ghihrhhordgtohhmqeenucggtffrrghtthgvrhhnpefffffhveffueegvefgieekvedthf evvedtieeljeeuieekleeiveeigefgffdujeenucffohhmrghinhepheenucfkphepjeeg rdduudeirddukeeirdeggeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpehsrghmphhlvghtsehnghihrhhordgtohhm X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA id 1FDA0240356 for ; Tue, 23 Mar 2021 00:42:14 -0400 (EDT) From: Timothy Sample To: bug-guix@gnu.org Subject: Disarchive as a fallback for downloads Date: Tue, 23 Mar 2021 00:42:12 -0400 Message-ID: <87eeg6o50b.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=64.147.123.19; envelope-from=samplet@ngyro.com; helo=wout3-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) Hello, This patch series adds Disarchive assembly (backed by SWH lookup) as a fallback for downloads. To try it, make sure you are running the daemon in an environment with Disarchive available: $ ./pre-inst-env guix environment --ad-hoc guile disarchive # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild Don=E2=80=99t forget to stop your existing Guix Daemon. :) You also need to make sure that regular downloads are unavailable. I do this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Curl= -fetch=E2=80=9D in =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of URL= s with =E2=80=98()=E2=80=99: (let try ((uri (append uri content-addressed-uris))) (match '() ; uri ...)) Now you can ask Guix for a recent .tar.gz source package: $ ./pre-inst-env guix build --no-substitutes -S python-httpretty You should see: Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g5v= diwlv19y9-httpretty-1.0.5.tar.gz Assembling the directory httpretty-1.0.5 Downloading from Software Heritage... 7903d608efc89c14afb4d692a3721156e31a43e2/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/ 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING [...] Checking httpretty-1.0.5 digest... ok Assembling the tarball httpretty-1.0.5.tar Checking httpretty-1.0.5.tar digest... ok Assembling the Gzip file httpretty-1.0.5.tar.gz Checking httpretty-1.0.5.tar.gz digest... ok Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty= -1.0.5.tar.gz successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httprett= y-1.0.5.tar.gz.drv There=E2=80=99s lots to talk about though.... First, it looks up the metadata on my server. This is fine for a demo, but not what we want forever. The patch series supports adding several mirrors for looking up the metadata. In the past, we talked about putting everything on one or a few of the big Git hosting platforms like GitHub or Gitlab. That way, it would be easily picked up by SWH and archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to bu= ild the metadata, and a little script that moves it from the build server to my Web server. It would be simple enough to adjust that script to push it to a remote Git repo. (Of course, the next step is to move this setup to Guix infrastructure.) Thoughts? On the code level, there were two things I couldn=E2=80=99t figure out for myself. I made the mirror list just simple strings. AIUI, the client and the daemon have to agree about the format of the mirror list. Given that running old daemons is common, changing the format is difficult. Is it worth it to copy the more flexible interface used by the content addressed mirrors? If yes, do I have to do the same =E2=80=98module-autolo= ad!=E2=80=99 dance to use =E2=80=98bytevector->base16-string=E2=80=99? :) (I probably = would have just copied it, but that part confused me a bit.) I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (wel= l, just =E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D). It feels weird to use= a bunch of host-side modules from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module. This = is okay because =E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code. = It=E2=80=99s more like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build=E2= =80=9D, which is why importing host-side modules is OK... right? Hopefully everything else is more-or-less fine. :) -- Tim ------------=_1619717102-24882-1-- From unknown Sat Aug 09 13:19:09 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#47336] Disarchive as a fallback for downloads Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 14 May 2021 21:37:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47336 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Timothy Sample Cc: 47336@debbugs.gnu.org Received: via spool by 47336-submit@debbugs.gnu.org id=B47336.162102819116650 (code B ref 47336); Fri, 14 May 2021 21:37:01 +0000 Received: (at 47336) by debbugs.gnu.org; 14 May 2021 21:36:31 +0000 Received: from localhost ([127.0.0.1]:47215 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhfTz-0004KU-2x for submit@debbugs.gnu.org; Fri, 14 May 2021 17:36:31 -0400 Received: from eggs.gnu.org ([209.51.188.92]:57806) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhfTx-0004KH-5j for 47336@debbugs.gnu.org; Fri, 14 May 2021 17:36:30 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:45334) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhfTr-0004QY-Si; Fri, 14 May 2021 17:36:23 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49598 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhfTr-0002P0-I8; Fri, 14 May 2021 17:36:23 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87eeg6o50b.fsf@ngyro.com> Date: Fri, 14 May 2021 23:36:21 +0200 In-Reply-To: <87eeg6o50b.fsf@ngyro.com> (Timothy Sample's message of "Tue, 23 Mar 2021 00:42:12 -0400") Message-ID: <87im3lknfu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Timothy Sample skribis: > This patch series adds Disarchive assembly (backed by SWH lookup) as a > fallback for downloads. > > To try it, make sure you are running the daemon in an environment with > Disarchive available: > > $ ./pre-inst-env guix environment --ad-hoc guile disarchive > # ./pre-inst-env guix-daemon --build-users-group=3Dguixbuild > > Don=E2=80=99t forget to stop your existing Guix Daemon. :) > > You also need to make sure that regular downloads are unavailable. I do > this by adjusting the =E2=80=9Ctry=E2=80=9D loop at the end of =E2=80=9Cu= rl-fetch=E2=80=9D in > =E2=80=9Cguix/build/download.scm=E2=80=9D. I replace the usual list of U= RLs with =E2=80=98()=E2=80=99: > > (let try ((uri (append uri content-addressed-uris))) > (match '() ; uri > ...)) > > Now you can ask Guix for a recent .tar.gz source package: > > $ ./pre-inst-env guix build --no-substitutes -S python-httpretty > > You should see: > > Trying to use Disarchive to assemble /gnu/store/kbcnm57y2q1jvhvd8zw1g= 5vdiwlv19y9-httpretty-1.0.5.tar.gz > Assembling the directory httpretty-1.0.5 > Downloading from Software Heritage... > 7903d608efc89c14afb4d692a3721156e31a43e2/ > 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/ > 7903d608efc89c14afb4d692a3721156e31a43e2/httpretty-1.0.5/COPYING > [...] > Checking httpretty-1.0.5 digest... ok > Assembling the tarball httpretty-1.0.5.tar > Checking httpretty-1.0.5.tar digest... ok > Assembling the Gzip file httpretty-1.0.5.tar.gz > Checking httpretty-1.0.5.tar.gz digest... ok > Copying result to /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpret= ty-1.0.5.tar.gz > successfully built /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpre= tty-1.0.5.tar.gz.drv Commits 67bf61255414115ffae0141df9dd3623bc742bff and 0b1f70d1a792af40aa0d13b3d227fde88f02d061 add the dependency on Disarchive, so this fallback path is now enabled! > There=E2=80=99s lots to talk about though.... > > First, it looks up the metadata on my server. This is fine for a demo, > but not what we want forever. The patch series supports adding several > mirrors for looking up the metadata. In the past, we talked about > putting everything on one or a few of the big Git hosting platforms like > GitHub or Gitlab. That way, it would be easily picked up by SWH and > archived =E2=80=9Cforever=E2=80=9D. Right now, I have Cuirass set up to = build the > metadata, and a little script that moves it from the build server to my > Web server. It would be simple enough to adjust that script to push it > to a remote Git repo. (Of course, the next step is to move this setup > to Guix infrastructure.) Thoughts? We should talk to SWH, giving them the figures you gave earlier in this thread. But yeah, a Git repo looks best to me (it would be useful to keep track of changes, for example if we eventually update metadata to a new format) and it simplifies archival to SWH. Second thing we need to figure out if where to create this database. If you have a Cuirass job already, we should run it on ci.guix. WDYT? > On the code level, there were two things I couldn=E2=80=99t figure out for > myself. > > I made the mirror list just simple strings. AIUI, the client and the > daemon have to agree about the format of the mirror list. Given that > running old daemons is common, changing the format is difficult. Is it > worth it to copy the more flexible interface used by the content > addressed mirrors? If yes, do I have to do the same =E2=80=98module-auto= load!=E2=80=99 > dance to use =E2=80=98bytevector->base16-string=E2=80=99? :) (I probabl= y would have > just copied it, but that part confused me a bit.) I had overlooked this suggestion of yours. Yes, I think it=E2=80=99s best = to copy the SWH scheme. Don=E2=80=99t worry about =E2=80=98module-autoload!= =E2=80=99: nowadays we can safely assume (guix base16) is available. When we change from list-of-strings to list-of-procedures, we=E2=80=99ll ha= ve to adjust the (guix build download) code so that it can deal with both. > I imported some modules from =E2=80=9Cguix/build/download.scm=E2=80=9D (w= ell, just > =E2=80=9Cbase16=E2=80=9D and =E2=80=9Cswh=E2=80=9D). It feels weird to u= se a bunch of host-side modules > from what=E2=80=99s nominally a =E2=80=9Cguix/build=E2=80=9D module. Thi= s is okay because > =E2=80=9Cguix/build/download.scm=E2=80=9D is not /really/ build-side code= . It=E2=80=99s more > like daemon (-ish) code that just happens to live in =E2=80=9Cguix/build= =E2=80=9D, which > is why importing host-side modules is OK... right? Yup. :-) In the end, the whole point is to reuse code on both sides, and that=E2=80=99s what=E2=80=99s being done here. Thanks, Ludo=E2=80=99.