From debbugs-submit-bounces@debbugs.gnu.org Sat May 15 06:28:25 2021 Received: (at submit) by debbugs.gnu.org; 15 May 2021 10:28:25 +0000 Received: from localhost ([127.0.0.1]:47682 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhrWy-0001yG-Se for submit@debbugs.gnu.org; Sat, 15 May 2021 06:28:25 -0400 Received: from lists.gnu.org ([209.51.188.17]:38636) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhrWx-0001y9-9m for submit@debbugs.gnu.org; Sat, 15 May 2021 06:28:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37250) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lhrWx-0003iH-3D for guix-patches@gnu.org; Sat, 15 May 2021 06:28:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40358) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lhrWw-0004sS-PL; Sat, 15 May 2021 06:28:22 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51246 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lhrWw-00037e-G3; Sat, 15 May 2021 06:28:22 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH] lint: archival: Lookup content in Disarchive database. Date: Sat, 15 May 2021 12:28:14 +0200 Message-Id: <20210515102814.5944-1-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: Timothy Sample , =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/lint.scm (lookup-disarchive-spec): New procedure. (check-archival): When 'lookup-content' returns #f, call 'lookup-disarchive-spec'. * guix/download.scm (%disarchive-mirrors): Make public. --- guix/download.scm | 1 + guix/lint.scm | 31 +++++++++++++++++++++++++++---- 2 files changed, 28 insertions(+), 4 deletions(-) Hello! This patch makes the ‘archival’ checker check the Disarchive database(s) when SWH ‘lookup-content’ returns #f. For example, before the patch, we get: $ guix lint -c archival guile-json gnu/packages/guile.scm:622:12: guile-json@4.5.2: source not archived on Software Heritage After the patch, we get nothing (success) thanks to Disarchive metadata available at: https://disarchive.ngyro.com/sha256/1ab046ec36b1c44c041ac275568d818784d71fab9a5d95f9128cfe8a25051933 It assumes that the swhid found in the Disarchive metadata is valid, a reasonable assumption IMO. Thoughts? Ludo’. diff --git a/guix/download.scm b/guix/download.scm index 72094e7318..b6eb97e6fa 100644 --- a/guix/download.scm +++ b/guix/download.scm @@ -35,6 +35,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) #:export (%mirrors + %disarchive-mirrors (url-fetch* . url-fetch) url-fetch/executable url-fetch/tarbomb diff --git a/guix/lint.scm b/guix/lint.scm index 1bebfe03d3..c6ad54ddeb 100644 --- a/guix/lint.scm +++ b/guix/lint.scm @@ -30,6 +30,7 @@ (define-module (guix lint) #:use-module (guix store) + #:autoload (guix base16) (bytevector->base16-string) #:use-module (guix base32) #:use-module (guix diagnostics) #:use-module (guix download) @@ -1227,6 +1228,23 @@ upstream releases") #:field 'source))))))) +(define (lookup-disarchive-spec hash) + "Return true if Disarchive mirrors have a spec for HASH, false otherwise." + (any (lambda (mirror) + (with-networking-fail-safe + (format #f (G_ "failed to access Disarchive database at ~a") + mirror) + #f + (let* ((url (string-append mirror + (symbol->string + (content-hash-algorithm hash)) + "/" + (bytevector->base16-string + (content-hash-value hash)))) + (response (http-head url))) + (= 200 (response-code response))))) + %disarchive-mirrors)) + (define (check-archival package) "Check whether PACKAGE's source code is archived on Software Heritage. If it's not, and if its source code is a VCS snapshot, then send a \"save\" @@ -1302,10 +1320,15 @@ try again later") (symbol->string (content-hash-algorithm hash))) (#f - (list (make-warning package - (G_ "source not archived on Software \ -Heritage") - #:field 'source))) + ;; If SWH doesn't have HASH as is, it may be because it's + ;; a hand-crafted tarball. In that case, check whether + ;; the Disarchive database has an entry for that tarball. + (if (lookup-disarchive-spec hash) + '() + (list (make-warning package + (G_ "source not archived on Software \ +Heritage and missing from the Disarchive database") + #:field 'source)))) ((? content?) '()))) '())))) -- 2.31.1 From debbugs-submit-bounces@debbugs.gnu.org Mon May 17 23:19:11 2021 Received: (at submit) by debbugs.gnu.org; 18 May 2021 03:19:11 +0000 Received: from localhost ([127.0.0.1]:54200 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liqGF-0004K3-1m for submit@debbugs.gnu.org; Mon, 17 May 2021 23:19:11 -0400 Received: from lists.gnu.org ([209.51.188.17]:45744) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1liqGB-0004Ju-Pq for submit@debbugs.gnu.org; Mon, 17 May 2021 23:19:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48622) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1liqGB-00068s-Cm for guix-patches@gnu.org; Mon, 17 May 2021 23:19:07 -0400 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:60489) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1liqG9-0003ej-E7; Mon, 17 May 2021 23:19:07 -0400 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 6E5B35C0198; Mon, 17 May 2021 23:19:02 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Mon, 17 May 2021 23:19:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=4A570MIgOwm3cqyoO+6ZRegdsmoVkpka2dXrVAPD0 v0=; b=K4OFflyNF8icGdiY27Q3PxoM2ntLE+QPs3PAh2T8P6Lli0cNnKBww1KHb wq9PC7dGvX/88ILu6TATimo7uyt4s5xw8z5a0o4AXRI/6RkHcEp5xg8eGR+iyUCy j5mfAmPeUlRbaLVe0r3EndgeuzSppMK33C+P7bc+Q5kCDA93sFGDQo46Tm3BMHiN 1twkXlNDWn4uhqBiawH3kCG+AczuKLPuTHky0i8qigI2njSS/t5wcdx7x/b6kD33 BR9bOmrTIyPPY4cb26OVDymWJxEihLTBrMsEKj/weFdZM9km/q2XcZk2OBGs5gB8 d4fZdo9W7FVDG5SVBbn9hz+16ZP8w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdeiiedgieehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufhfffgjkfgfgggtgfesthhqredttderjeenucfhrhhomhepvfhimhho thhhhicuufgrmhhplhgvuceoshgrmhhplhgvthesnhhghihrohdrtghomheqnecuggftrf grthhtvghrnhepieduuefhgeegleelveehgedugfeuhfeikefftdevieelgfelhfdvtdfg ieehtefgnecukfhppeejgedrudduiedrudekiedrgeegnecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomhepshgrmhhplhgvthesnhhghihrohdrtgho mh X-ME-Proxy: Received: from mrblack (74-116-186-44.qc.dsl.ebox.net [74.116.186.44]) by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 17 May 2021 23:19:01 -0400 (EDT) From: Timothy Sample To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [PATCH] lint: archival: Lookup content in Disarchive database. References: <20210515102814.5944-1-ludo@gnu.org> Date: Mon, 17 May 2021 23:19:01 -0400 In-Reply-To: <20210515102814.5944-1-ludo@gnu.org> ("Ludovic =?utf-8?Q?Cour?= =?utf-8?Q?t=C3=A8s=22's?= message of "Sat, 15 May 2021 12:28:14 +0200") Message-ID: <87o8d8loey.fsf@ngyro.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=66.111.4.28; envelope-from=samplet@ngyro.com; helo=out4-smtp.messagingengine.com X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.6 (-) X-Debbugs-Envelope-To: submit Cc: guix-patches@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) Hello, Ludovic Court=C3=A8s writes: > This patch makes the =E2=80=98archival=E2=80=99 checker check the Disarch= ive database(s) > when SWH =E2=80=98lookup-content=E2=80=99 returns #f. [...] > > It assumes that the swhid found in the Disarchive metadata is valid, a > reasonable assumption IMO. > > Thoughts? One thing to consider is that just because Disarchive has captured an archive=E2=80=99s metadata and computed the SWHID of its contents doesn=E2= =80=99t mean that the contents are actually in the SWH archive. (Maybe that=E2=80=99s w= hat you meant when you wrote about valid IDs above.) It would be neat if the lint check looked up the SWHID to see if it exists. Unfortunately, Disarchive doesn=E2=80=99t make getting the underlying SWHID easy at the mo= ment. One option would be to pass a resolver to =E2=80=9Cdisarchive-assemble=E2= =80=9D that exfiltrates the ID using =E2=80=9Cset!=E2=80=9D. Another one would be to = =E2=80=9Cread=E2=80=9D the specification and search for a form like =E2=80=98(swhid "swh:1:dir:...")= =E2=80=99. Neither is particularly lovely.... Other than that, the code looks good and everything seems to work. :) -- Tim From debbugs-submit-bounces@debbugs.gnu.org Tue May 18 17:47:56 2021 Received: (at 48437) by debbugs.gnu.org; 18 May 2021 21:47:56 +0000 Received: from localhost ([127.0.0.1]:57499 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj7ZE-0006dg-5y for submit@debbugs.gnu.org; Tue, 18 May 2021 17:47:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41114) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lj7ZB-0006dU-RS for 48437@debbugs.gnu.org; Tue, 18 May 2021 17:47:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:53280) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lj7Z6-0005zT-Bi; Tue, 18 May 2021 17:47:48 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37520 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lj7Z5-0006cA-Ge; Tue, 18 May 2021 17:47:48 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Timothy Sample Subject: Re: bug#48437: [PATCH] lint: archival: Lookup content in Disarchive database. References: <20210515102814.5944-1-ludo@gnu.org> <87o8d8loey.fsf@ngyro.com> Date: Tue, 18 May 2021 23:47:45 +0200 In-Reply-To: <87o8d8loey.fsf@ngyro.com> (Timothy Sample's message of "Mon, 17 May 2021 23:19:01 -0400") Message-ID: <87a6or67em.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 48437 Cc: 48437@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Timothy Sample skribis: > Ludovic Court=C3=A8s writes: > >> This patch makes the =E2=80=98archival=E2=80=99 checker check the Disarc= hive database(s) >> when SWH =E2=80=98lookup-content=E2=80=99 returns #f. [...] >> >> It assumes that the swhid found in the Disarchive metadata is valid, a >> reasonable assumption IMO. >> >> Thoughts? > > One thing to consider is that just because Disarchive has captured an > archive=E2=80=99s metadata and computed the SWHID of its contents doesn= =E2=80=99t mean > that the contents are actually in the SWH archive. (Maybe that=E2=80=99s= what > you meant when you wrote about valid IDs above.) Yes, I thought we could assume the contents were necessarily in the archive. > It would be neat if the lint check looked up the SWHID to see if it > exists. Unfortunately, Disarchive doesn=E2=80=99t make getting the under= lying > SWHID easy at the moment. One option would be to pass a resolver to > =E2=80=9Cdisarchive-assemble=E2=80=9D that exfiltrates the ID using =E2= =80=9Cset!=E2=80=9D. Another > one would be to =E2=80=9Cread=E2=80=9D the specification and search for a= form like > =E2=80=98(swhid "swh:1:dir:...")=E2=80=99. Neither is particularly lovel= y.... Hmm yeah. There=E2=80=99s no API to deserialize the (disarchive =E2=80=A6)= sexp as a record, right? > Other than that, the code looks good and everything seems to work. :) Maybe we can assume (with a comment) that the SWHID points to a valid content, and when creating the Disarchive database we actually make sure this is the case? More generally, we need to talk about that database, how to create it and maintain it. :-) Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 06:27:20 2021 Received: (at 48437) by debbugs.gnu.org; 21 May 2021 10:27:20 +0000 Received: from localhost ([127.0.0.1]:35897 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lk2ND-0003Rl-L2 for submit@debbugs.gnu.org; Fri, 21 May 2021 06:27:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:57276) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lk2NA-0003RX-Nf for 48437@debbugs.gnu.org; Fri, 21 May 2021 06:27:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49118) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lk2N5-0003RZ-3g; Fri, 21 May 2021 06:27:11 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55948 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lk2N3-00023d-RD; Fri, 21 May 2021 06:27:10 -0400 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 48437@debbugs.gnu.org Subject: [PATCH v2] lint: archival: Lookup content in Disarchive database. Date: Fri, 21 May 2021 12:27:03 +0200 Message-Id: <20210521102703.23383-1-ludo@gnu.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <87a6or67em.fsf_-_@gnu.org> References: <87a6or67em.fsf_-_@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 48437 Cc: Timothy Sample , =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/lint.scm (lookup-disarchive-spec): New procedure. (check-archival): When 'lookup-content' returns #f, call 'lookup-disarchive-spec'. Call 'lookup-directory' on the result of 'lookup-directory'. * guix/download.scm (%disarchive-mirrors): Make public. * tests/lint.scm ("archival: missing content"): Set '%disarchive-mirrors'. ("archival: content unavailable but disarchive available"): New test. --- guix/download.scm | 1 + guix/lint.scm | 62 ++++++++++++++++++++++++++++++++++++++++++++--- tests/lint.scm | 34 +++++++++++++++++++++++--- 3 files changed, 89 insertions(+), 8 deletions(-) Hi! This new version checks that the SWH IDs that appear in a Disarchive entry are indeed available at archive.softwareheritage.org. It also adds a test for that. Ludo'. diff --git a/guix/download.scm b/guix/download.scm index 72094e7318..b6eb97e6fa 100644 --- a/guix/download.scm +++ b/guix/download.scm @@ -35,6 +35,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) #:export (%mirrors + %disarchive-mirrors (url-fetch* . url-fetch) url-fetch/executable url-fetch/tarbomb diff --git a/guix/lint.scm b/guix/lint.scm index 1bebfe03d3..a2d6418b85 100644 --- a/guix/lint.scm +++ b/guix/lint.scm @@ -30,6 +30,7 @@ (define-module (guix lint) #:use-module (guix store) + #:autoload (guix base16) (bytevector->base16-string) #:use-module (guix base32) #:use-module (guix diagnostics) #:use-module (guix download) @@ -1227,6 +1228,43 @@ upstream releases") #:field 'source))))))) +(define (lookup-disarchive-spec hash) + "If Disarchive mirrors have a spec for HASH, return the list of SWH +directory identifiers the spec refers to. Otherwise return #f." + (define (extract-swh-id spec) + ;; Return the list of SWH directory identifiers SPEC refers to, where SPEC + ;; is a Disarchive sexp. Instead of attempting to parse it, traverse it + ;; in a pretty unintelligent fashion. + (let loop ((sexp spec) + (ids '())) + (match sexp + ((? string? str) + (let ((prefix "swh:1:dir:")) + (if (string-prefix? prefix str) + (cons (string-drop str (string-length prefix)) ids) + ids))) + ((head tail ...) + (loop tail (loop head ids))) + (_ ids)))) + + (any (lambda (mirror) + (with-networking-fail-safe + (format #f (G_ "failed to access Disarchive database at ~a") + mirror) + #f + (guard (c ((http-get-error? c) #f)) + (let* ((url (string-append mirror + (symbol->string + (content-hash-algorithm hash)) + "/" + (bytevector->base16-string + (content-hash-value hash)))) + (port (http-fetch (string->uri url) #:text? #t)) + (spec (read port))) + (close-port port) + (extract-swh-id spec))))) + %disarchive-mirrors)) + (define (check-archival package) "Check whether PACKAGE's source code is archived on Software Heritage. If it's not, and if its source code is a VCS snapshot, then send a \"save\" @@ -1302,10 +1340,26 @@ try again later") (symbol->string (content-hash-algorithm hash))) (#f - (list (make-warning package - (G_ "source not archived on Software \ -Heritage") - #:field 'source))) + ;; If SWH doesn't have HASH as is, it may be because it's + ;; a hand-crafted tarball. In that case, check whether + ;; the Disarchive database has an entry for that tarball. + (match (lookup-disarchive-spec hash) + (#f + (list (make-warning package + (G_ "source not archived on Software \ +Heritage and missing from the Disarchive database") + #:field 'source))) + (directory-ids + (match (find (lambda (id) + (not (lookup-directory id))) + directory-ids) + (#f '()) + (id + (list (make-warning package + (G_ " +Disarchive entry refers to non-existent SWH directory '~a'") + (list id) + #:field 'source))))))) ((? content?) '()))) '())))) diff --git a/tests/lint.scm b/tests/lint.scm index a2c8665142..d54fafc1d2 100644 --- a/tests/lint.scm +++ b/tests/lint.scm @@ -1,7 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013 Cyril Roelandt ;;; Copyright © 2014, 2015, 2016 Eric Bavier -;;; Copyright © 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2015, 2016 Mathieu Lirzin ;;; Copyright © 2016 Hartmut Goebel ;;; Copyright © 2017 Alex Kost @@ -1008,10 +1008,13 @@ (method url-fetch) (uri "http://example.org/foo.tgz") (sha256 (make-bytevector 32)))) - (warnings (with-http-server '((404 "Not archived.")) + (warnings (with-http-server '((404 "Not archived.") + (404 "Not in Disarchive database.")) (parameterize ((%swh-base-url (%local-url))) - (check-archival (dummy-package "x" - (source origin))))))) + (mock ((guix download) %disarchive-mirrors + (list (%local-url))) + (check-archival (dummy-package "x" + (source origin)))))))) (warning-contains? "not archived" warnings))) (test-equal "archival: content available" @@ -1027,6 +1030,29 @@ (parameterize ((%swh-base-url (%local-url))) (check-archival (dummy-package "x" (source origin))))))) +(test-equal "archival: content unavailable but disarchive available" + '() + (let* ((origin (origin + (method url-fetch) + (uri "http://example.org/foo.tgz") + (sha256 (make-bytevector 32)))) + (disarchive (object->string + '(disarchive (version 0) + ... + "swh:1:dir:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"))) + ;; https://archive.softwareheritage.org/api/1/directory/ + (directory "[ { \"checksums\": {}, + \"dir_id\": \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\", + \"type\": \"file\", + \"name\": \"README\" + \"length\": 42 } ]")) + (with-http-server `((404 "") ;lookup-content + (200 ,disarchive) ;Disarchive database lookup + (200 ,directory)) ;lookup-directory + (mock ((guix download) %disarchive-mirrors (list (%local-url))) + (parameterize ((%swh-base-url (%local-url))) + (check-archival (dummy-package "x" (source origin)))))))) + (test-assert "archival: missing revision" (let* ((origin (origin (method git-fetch) -- 2.31.1 From debbugs-submit-bounces@debbugs.gnu.org Sat May 22 17:52:39 2021 Received: (at 48437-done) by debbugs.gnu.org; 22 May 2021 21:52:39 +0000 Received: from localhost ([127.0.0.1]:40211 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lkZXz-0004ZE-DN for submit@debbugs.gnu.org; Sat, 22 May 2021 17:52:39 -0400 Received: from eggs.gnu.org ([209.51.188.92]:54786) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lkZXx-0004Z1-Qg for 48437-done@debbugs.gnu.org; Sat, 22 May 2021 17:52:38 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:40938) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lkZXs-00055U-Ei; Sat, 22 May 2021 17:52:32 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55782 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lkZXs-0006Il-6V; Sat, 22 May 2021 17:52:32 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 48437-done@debbugs.gnu.org Subject: Re: bug#48437: [PATCH] lint: archival: Lookup content in Disarchive database. References: <87a6or67em.fsf_-_@gnu.org> <20210521102703.23383-1-ludo@gnu.org> Date: Sat, 22 May 2021 23:52:31 +0200 In-Reply-To: <20210521102703.23383-1-ludo@gnu.org> ("Ludovic =?utf-8?Q?Cou?= =?utf-8?Q?rt=C3=A8s=22's?= message of "Fri, 21 May 2021 12:27:03 +0200") Message-ID: <87im3a5tcw.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 48437-done Cc: Timothy Sample X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Ludovic Court=C3=A8s skribis: > * guix/lint.scm (lookup-disarchive-spec): New procedure. > (check-archival): When 'lookup-content' returns #f, call > 'lookup-disarchive-spec'. Call 'lookup-directory' on the result of > 'lookup-directory'. > * guix/download.scm (%disarchive-mirrors): Make public. > * tests/lint.scm ("archival: missing content"): Set > '%disarchive-mirrors'. > ("archival: content unavailable but disarchive available"): New test. Following our discussion on IRC, I pushed this variant as bc4d81d267830a3b1ccb63198f4100cc836e4e4e. Thanks for taking a look! Ludo=E2=80=99. From unknown Tue Jun 17 01:48:34 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 20 Jun 2021 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator