From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 23 11:13:40 2020 Received: (at submit) by debbugs.gnu.org; 23 Jun 2020 15:13:40 +0000 Received: from localhost ([127.0.0.1]:36960 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnkcF-0006G6-SV for submit@debbugs.gnu.org; Tue, 23 Jun 2020 11:13:40 -0400 Received: from lists.gnu.org ([209.51.188.17]:59120) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnkcD-0006Fy-Ei for submit@debbugs.gnu.org; Tue, 23 Jun 2020 11:13:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42196) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jnkcD-00049r-7x for guix-patches@gnu.org; Tue, 23 Jun 2020 11:13:37 -0400 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]:34889) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jnkcA-00070Y-R5 for guix-patches@gnu.org; Tue, 23 Jun 2020 11:13:36 -0400 Received: by mail-wm1-x32d.google.com with SMTP id g21so3645629wmg.0 for ; Tue, 23 Jun 2020 08:13:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dGcudQtIfd3qOwxYBWEUb8CHVMojlWlT7ckROFvsylk=; b=nnI8PSQsnkTBXn9h8wNFMmZIT2y4vtPMh17VQnrbNcVWQWNtdfTOluVrApzIvABCx6 ATwLInvZ9b3HCUqHSp3WZWladOOSzKJdnCz43IbXM9jbz8xTFvMlSGVDBBlnj4byFskH +pTC/FvEMO0KiXV8oQyXE2PBX9wFGe31ag/BYantr6dk7ZllGql1LjF4z1TCnqnWgFqr 3PvJvrcikORhhGoZM7exHkUjKqSrtcZS8h0nmXpUtfRHYEXk46IL2XFrAUPI50w0shDF Z2cxvWDktr2MgmM/l3E3we5KOp4jxljO8clMdCxmBuk5wMJjaaivwj5q/WAOtqKH7cjW J8vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dGcudQtIfd3qOwxYBWEUb8CHVMojlWlT7ckROFvsylk=; b=VHatpyIOPToRP9DEoHK7LKf1QQK69nTKfcNmVuLj86Gcp4Uhg8Nb6HWjXXBK90fdz0 TTNtNVusc02GC8gKUFCGCkhNwYfVU7epQgUA1hR8WhtzCNDFlFzi9dRRKXfZV2UDdLub BVimMLLeef5IfNNZc+Z5CuKw9VN44iSPDXwu8NV102z2KyN2OPyUVgu3eJxDkzxCOTU+ R2Co5ujW91q9ixC373cZdWhXjvwPvJDcih2ShId5+84MipqsqKMRLh5pZLE3yOqhdJwS vQXMjZd5RX9hm2SOdZnCsMt72aarp/fUek44n+Ck2xQhdPHcBy/DDmRP53OaclSx8t1e OrxA== X-Gm-Message-State: AOAM533tBgjbVX9bUs6qPqrnHNnZGSaUWhD7G5NHow9Ux4YqHrZaYL6l LDiZ6XEM+IwH00Fyeo2ReegPd48Zr1k= X-Google-Smtp-Source: ABdhPJxni9NVvpKLahFnioUCuTG42rRyGWyXCE4I+QJTvRjbfoF9YkAK6JmNDw/KeBbzFdUvLPU7RQ== X-Received: by 2002:a1c:9802:: with SMTP id a2mr23657967wme.64.1592925212844; Tue, 23 Jun 2020 08:13:32 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id n8sm19630726wrj.44.2020.06.23.08.13.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jun 2020 08:13:32 -0700 (PDT) From: zimoun To: guix-patches@gnu.org Subject: [PATCH 0/1] sources.json compliant with SWH loader Date: Tue, 23 Jun 2020 17:13:23 +0200 Message-Id: <20200623151323.29639-1-zimon.toutoune@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2a00:1450:4864:20::32d; envelope-from=zimon.toutoune@gmail.com; helo=mail-wm1-x32d.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit Cc: zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Dear, This patch adds the "integrity" field. It is SRI format i.e., 'origin-hash' is converted to 'base64'. The "revision" field is the Guix commit. It should be used by SWH; for example SWH could fetch several sources.json. Currently, the SWH loader does only support the formats [1] ".tar.gz$|.zip$|tar.bz2$|.tbz$|.tar.xz$|.tgz$|.tar$" and their advice is to filter out any other files (e.g., Gem). For now, there is no filter and it could be added then if it is really an issue for them. 1: https://forge.softwareheritage.org/T1352#45459 All the best, simon zimoun (1): website: Add integrity to JSON sources. website/apps/packages/builder.scm | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) base-commit: 36fdde5b3efad445291588a5bc17a11802eb7ff8 -- 2.26.2 From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 23 11:21:52 2020 Received: (at 42019) by debbugs.gnu.org; 23 Jun 2020 15:21:52 +0000 Received: from localhost ([127.0.0.1]:36965 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnkkB-0006Sa-Ph for submit@debbugs.gnu.org; Tue, 23 Jun 2020 11:21:52 -0400 Received: from mail-wr1-f43.google.com ([209.85.221.43]:42376) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnkk9-0006SN-KH for 42019@debbugs.gnu.org; Tue, 23 Jun 2020 11:21:50 -0400 Received: by mail-wr1-f43.google.com with SMTP id o11so13244368wrv.9 for <42019@debbugs.gnu.org>; Tue, 23 Jun 2020 08:21:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=O+lryE6sgzmhEzUsnEWC9eVcX9quyQkDEHN/amD0iMo=; b=q6jUr8RefnjPdSaLqypM+dC+LOKj+ixFm1sLBAcuaNU5HZ6J7wwy47tXHw45yMenxh 6AmnR6x/ZxsEfmAdXZxT5oXlBpkOdRzY4Tg6TYA6FfZOtomt5RxFzohJ+ySfZRRXSFCO aX+B8DW90mF/pDH8UAhU9BTvtuIVDUP8w9zZZ2LO1JE2giHkIavXX4dp3zZcQvEroS1f V0Bqr2xcBvR/adcxxHfWwGSeLAhie78iWHVYyNOQlH6wrCW32oeLW61XsKiawhMvMAt4 UTd8aM5xRWYk52gN0cwj9pWAbNnA676eS40GsJohPZw0iBF8ziMP0rwalvawJBd9aYQq de0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=O+lryE6sgzmhEzUsnEWC9eVcX9quyQkDEHN/amD0iMo=; b=iY56KC17Q5lGvo3iUcYezia1LWt9KvNQaI9c1zFg5Rkj9jmEWoF2+hV+049dwXQK/R 08bx3FG5/rdBolXJ98OaTf76KhZFDtmDUm/SDa8u59QQ5yMfjkgPo68ZSkSKN8fzeJ1x pha4UkShgkXLjzbsu38GkOrMBRcITwzQ7gPXjAvYTu+HK8RYH6V4OlUgls915pAUTlDC e5qV6smYuCBVlygZmNA2TOzmhI018RkH7p9rCVbis9I4n/zW0RmBAlMoWP+gSVYaFWiT St46y8shRIMg3/W6aCgG2JTzhIXhTWED0n/H7Z9VVXX6mnFrS9HB5FqWBMWdZgMftvop JCIg== X-Gm-Message-State: AOAM5321Cg+7XQas/6A/74cZVYb3jTmflVsdSKxFEDQquAmmBSkitTRU 3bAy6EdW6NI0dDiPVH3RRWL9PE7/Xao= X-Google-Smtp-Source: ABdhPJz7obllbnAH3vgAM0Jjk85jGmQd4iLTrFqP2/aoh0y6wc+JSVbWPEgYF4+zQoAhag9Q/+gcpQ== X-Received: by 2002:a5d:4687:: with SMTP id u7mr10798938wrq.357.1592925703291; Tue, 23 Jun 2020 08:21:43 -0700 (PDT) Received: from localhost.localdomain ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id c143sm4003961wmd.1.2020.06.23.08.21.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jun 2020 08:21:42 -0700 (PDT) From: zimoun To: 42019@debbugs.gnu.org Subject: [PATCH 1/1] website: Add integrity to JSON sources. Date: Tue, 23 Jun 2020 17:21:39 +0200 Message-Id: <20200623152139.512-1-zimon.toutoune@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * website/apps/packages/builder.scm (origin->json): Add integrity field using SRI format. --- website/apps/packages/builder.scm | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm index d2bccd7..e20d672 100644 --- a/website/apps/packages/builder.scm +++ b/website/apps/packages/builder.scm @@ -46,6 +46,8 @@ #:use-module (guix hg-download) #:use-module (guix utils) ;location #:use-module ((guix build download) #:select (maybe-expand-mirrors)) + #:use-module ((guix base64) #:select (base64-encode)) + #:use-module ((guix config) #:select (%guix-version)) #:use-module (json) #:use-module (ice-9 match) #:use-module ((web uri) #:select (string->uri uri->string)) @@ -114,7 +116,7 @@ ,@(cond ((or (eq? url-fetch method) (eq? url-fetch/tarbomb method) (eq? url-fetch/zipbomb method)) - `(("url" . ,(list->vector + `(("urls" . ,(list->vector (resolve (match uri ((? string? url) (list url)) @@ -128,6 +130,16 @@ ((eq? hg-fetch method) `(("hg_url" . ,(hg-reference-url uri)))) (else '())) + ,@(if (or (eq? url-fetch method) + (eq? url-fetch/tarbomb method) + (eq? url-fetch/zipbomb method)) + (let* ((content-hash (origin-hash origin)) + (hash-value (content-hash-value content-hash)) + (hash-algorithm (content-hash-algorithm content-hash)) + (algorithm-string (symbol->string hash-algorithm))) + `(("integrity" . ,(string-append algorithm-string "-" + (base64-encode hash-value))))) + '()) ,@(if (eq? method git-fetch) `(("git_ref" . ,(git-reference-commit uri))) '()) @@ -174,9 +186,11 @@ scm->json)) (define (sources-json-builder) - "Return a JSON page listing all the sources. - -See ." + "Return a JSON page listing all the sources." + ;; The Software Heritage format is described here: + ;; https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources.json + ;; And the loader is implemented here: + ;; https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/package/nixguix/ (define (package->json package) `(,@(if (origin? (package-source package)) (origin->json (package-source package)) @@ -185,7 +199,8 @@ See ." (make-page "sources.json" `(("sources" . ,(list->vector (map package->json (all-packages)))) - ("version" . "1")) + ("version" . "1") + ("revision" . ,%guix-version)) scm->json)) (define (index-builder) -- 2.26.2 From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 13:06:01 2020 Received: (at 42019) by debbugs.gnu.org; 27 Jun 2020 17:06:01 +0000 Received: from localhost ([127.0.0.1]:44506 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEHB-00082X-42 for submit@debbugs.gnu.org; Sat, 27 Jun 2020 13:06:01 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47620) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEH8-00082C-NF for 42019@debbugs.gnu.org; Sat, 27 Jun 2020 13:06:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:59183) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jpEH3-0006Ux-H1; Sat, 27 Jun 2020 13:05:53 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=37600 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jpEH2-0000W5-OC; Sat, 27 Jun 2020 13:05:53 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> Date: Sat, 27 Jun 2020 19:05:50 +0200 In-Reply-To: <20200623152139.512-1-zimon.toutoune@gmail.com> (zimoun's message of "Tue, 23 Jun 2020 17:21:39 +0200") Message-ID: <87zh8ol9qp.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 42019 Cc: 42019@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! zimoun skribis: > * website/apps/packages/builder.scm (origin->json): Add integrity field u= sing > SRI format. [...] > - `(("url" . ,(list->vector > + `(("urls" . ,(list->vector > (resolve > (match uri > ((? string? url) (list url)) Is this change OK for Repology? Or should we keep =E2=80=9Curl=E2=80=9D in= addition to =E2=80=9Curls=E2=80=9D? > (make-page "sources.json" > `(("sources" . ,(list->vector (map package->json (all-packa= ges)))) > - ("version" . "1")) > + ("version" . "1") > + ("revision" . ,%guix-version)) There=E2=80=99s no guarantee that =E2=80=98%guix-version=E2=80=99 is a comm= it ID, so perhaps we should do something like: (match (current-profile) (#f %guix-version) ;for lack of a better ID (profile (let ((channel (find guix-channel? (profile-channels profile)))) (channel-commit channel)))) Otherwise LGTM, thank you! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 13:42:09 2020 Received: (at 42019) by debbugs.gnu.org; 27 Jun 2020 17:42:09 +0000 Received: from localhost ([127.0.0.1]:44558 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEq9-0000TJ-A2 for submit@debbugs.gnu.org; Sat, 27 Jun 2020 13:42:09 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:33149) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEq7-0000Sr-6A for 42019@debbugs.gnu.org; Sat, 27 Jun 2020 13:42:08 -0400 Received: by mail-wm1-f66.google.com with SMTP id a6so10942494wmm.0 for <42019@debbugs.gnu.org>; Sat, 27 Jun 2020 10:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=qPuDV0oiO2Lff4JNFtmchB9WvHsSrIowzCY8rbKPYIg=; b=dZr5rSUeR1uYhlIyRwmxVgDWmzcWvxv/OIyqCHRLYrRSZPHqqthZBF991a7DyLV91N E7LCwYEomu+ptEaw5ApRTr2YnJH/Gz2Z0mIIdsq49yKppO1LeSTpYLzDR2qrvj8X9HE5 EAAdXAMaOor7+f87rCwyDqSfeDZho9kfqinYwplK+2sD2WPpCxAYXXA0R/0OxAtdFznd eq76TiwASfGOdYQ661GYCaLvYztXT9c5X1daGR4suVyQiBnkBUa4OkuhZFXphwhmJoEO KGiIoKQNOLSHdZEqv5Uoii3wKzHo+r2IaqiBTXZYB9ckY6kyx8e0g4uAx6qHoAkvyd4D XOHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=qPuDV0oiO2Lff4JNFtmchB9WvHsSrIowzCY8rbKPYIg=; b=q2Rw4yLPNHkw2LPQuVm7XXPPlMnc2GqZPk1TCCIlT4xRvewNe46e4ss2KG4m46tLDR jSfHpa+tfDzu6MsX5PLzZhsDyZYmQAZPTk0exuGTS21uRmo4+ctfvWi+Ig5oHHF75Yep G1+s7Ri0ycuJHVMn3zD36EaLgvW2Yn3dl4Vg5ok9UaKiON/IAU8QKK/b510fjDbNNZEW WE8rBZ4mkzreO/vDMydZVTE+MlLcXHDoaeVaMpAqOfpTGteorMgTjUG75AiNt/Wyi5Qs L9OlFPv0wN3JXAzxoY0d1VcBgeIaGifMrdhGZBS3hAEWVzWvm/TtZAiM7nuKPRzEDL+Q LjFA== X-Gm-Message-State: AOAM531yrArCeJ9bO1cMZtHpy5ApgQ5fHlMlE5I6PksGE/9ywO4XXoR5 pG0SShSXvPV6QkYIaYVh/U93yLH6X/U= X-Google-Smtp-Source: ABdhPJwxMoMu+GLecaVSiQwiOSqgg13K5itz8WO20vdya5AmvKO7Q+WSWEuP/ZCe/hGk4wB2ZmwNiw== X-Received: by 2002:a1c:154:: with SMTP id 81mr8912424wmb.23.1593279720913; Sat, 27 Jun 2020 10:42:00 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id r3sm10553106wmh.36.2020.06.27.10.42.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 27 Jun 2020 10:42:00 -0700 (PDT) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. In-Reply-To: <87zh8ol9qp.fsf@gnu.org> References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> <87zh8ol9qp.fsf@gnu.org> Date: Sat, 27 Jun 2020 19:41:59 +0200 Message-ID: <86y2o8be3c.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: 42019@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, Thank you for the review. On Sat, 27 Jun 2020 at 19:05, Ludovic Court=C3=A8s wrote: >> - `(("url" . ,(list->vector >> + `(("urls" . ,(list->vector >> (resolve >> (match uri >> ((? string? url) (list url)) > > Is this change OK for Repology? Or should we keep =E2=80=9Curl=E2=80=9D = in addition to > =E2=80=9Curls=E2=80=9D? >From what I understood of their API [1] when I checked it, I may say yes. := -) Well, I do not think that repology parses the field 'sources'. 1: https://repology.org/addrepo > There=E2=80=99s no guarantee that =E2=80=98%guix-version=E2=80=99 is a co= mmit ID, so perhaps we > should do something like: Thanks for the tip, I did not know. I will sent a v2 with your suggestion or feel free to update the patch and push it. :-) Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 29 12:51:15 2020 Received: (at 42019) by debbugs.gnu.org; 29 Jun 2020 16:51:15 +0000 Received: from localhost ([127.0.0.1]:49267 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpwzz-000272-5G for submit@debbugs.gnu.org; Mon, 29 Jun 2020 12:51:15 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:33244) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpwzv-00026d-8F for 42019@debbugs.gnu.org; Mon, 29 Jun 2020 12:51:13 -0400 Received: by mail-wm1-f67.google.com with SMTP id a6so14128329wmm.0 for <42019@debbugs.gnu.org>; Mon, 29 Jun 2020 09:51:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=5SCBqqoJ4Knoxv/Ea/p3QLx6Zw6XclkwWNTl4hmBnLY=; b=ivbgHke7XPcmpcl8lSySk4k3no/Ocp8yQOD+7pd7Ro10n9cE4wGhSEgNNa3aIjV5B5 EAX9AnyroRZwd7A5lEBfb1REJCBq9RnAmSN4OChT1D73sbHB1vIdE/0EoTM8lwBNqVF9 tQYHwyy/yuzSDBWFCADNZzE6TkgeYLcor5dnzaGx9fKdIJ934ZXapo3I2+HE5bYGuU5S IROM2wL9ESM+m1jn8T46jcUHn5tJlLzb13FQOTza5VjDAdpv7w7ztJIOHvTOcFCLofhH NICWGA0OfomthPmxxBA8oEjkNogsIMwR8LUDFNdKMWE9ckfMtqzCYa5UgR8z0LUmT0j7 Xrtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=5SCBqqoJ4Knoxv/Ea/p3QLx6Zw6XclkwWNTl4hmBnLY=; b=naSjZdxmGrhYOp43RB+gJpJ+F+2Wd/vajCew79nWOdoBr+rs3bkjb/hDnH6z3hVRxV 5H/y6CBSmAE7qkJirynL0FQf0t6Cd1J0DyH93dBCpv1jChkgGOgZh3f2mtYImLKeJqDY ssm1215a7UwJ0BIKYvoy7e/+Nk/Y4k1hnBvi4WRXKfS3mXaXv+nnNHrUnO0j3A3A1eqa SLRRZUsA9m0f3zV5MNQHhFe+Zkv5yupRTdXFmrMiFnJKXQ2WzB5+JeB7fzprW5ht2Aa6 vggIKTVbXTRR+k9q8I7M725QzCCqS/4PVuvXv1DUS/B4g58gBLvl+ciLtwU0aR1Y/3AT yYGw== X-Gm-Message-State: AOAM533T5nXnJmH0z/sDyqgi2ttApjoGl0RL129SkBA6eJbf3M0rL62v 4VQLTi4GI7XDm+eYTGsJD5uXbuMTfX8= X-Google-Smtp-Source: ABdhPJx+ryNl6H6a6Fm9Z8yNwT6tBAXq3pFcAUx1CVoTyvueznJ1ycQ12HxfdZdWnKitfTWhnErwVw== X-Received: by 2002:a1c:154:: with SMTP id 81mr17104468wmb.23.1593449465043; Mon, 29 Jun 2020 09:51:05 -0700 (PDT) Received: from pfiuh02.univ-paris-diderot.fr ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id e5sm440731wrs.33.2020.06.29.09.51.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jun 2020 09:51:04 -0700 (PDT) From: zimoun To: 42019@debbugs.gnu.org Subject: [PATCH v2] website: Add integrity to JSON sources. Date: Mon, 29 Jun 2020 18:50:57 +0200 Message-Id: <20200629165057.9451-1-zimon.toutoune@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * website/apps/packages/builder.scm (origin->json): Add integrity field using SRI format. --- website/apps/packages/builder.scm | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/website/apps/packages/builder.scm b/website/apps/packages/builder.scm index d2bccd7..fa488a5 100644 --- a/website/apps/packages/builder.scm +++ b/website/apps/packages/builder.scm @@ -46,6 +46,9 @@ #:use-module (guix hg-download) #:use-module (guix utils) ;location #:use-module ((guix build download) #:select (maybe-expand-mirrors)) + #:use-module ((guix base64) #:select (base64-encode)) + #:use-module ((guix describe) #:select (current-profile)) + #:use-module ((guix config) #:select (%guix-version)) #:use-module (json) #:use-module (ice-9 match) #:use-module ((web uri) #:select (string->uri uri->string)) @@ -114,7 +117,7 @@ ,@(cond ((or (eq? url-fetch method) (eq? url-fetch/tarbomb method) (eq? url-fetch/zipbomb method)) - `(("url" . ,(list->vector + `(("urls" . ,(list->vector (resolve (match uri ((? string? url) (list url)) @@ -128,6 +131,16 @@ ((eq? hg-fetch method) `(("hg_url" . ,(hg-reference-url uri)))) (else '())) + ,@(if (or (eq? url-fetch method) + (eq? url-fetch/tarbomb method) + (eq? url-fetch/zipbomb method)) + (let* ((content-hash (origin-hash origin)) + (hash-value (content-hash-value content-hash)) + (hash-algorithm (content-hash-algorithm content-hash)) + (algorithm-string (symbol->string hash-algorithm))) + `(("integrity" . ,(string-append algorithm-string "-" + (base64-encode hash-value))))) + '()) ,@(if (eq? method git-fetch) `(("git_ref" . ,(git-reference-commit uri))) '()) @@ -174,9 +187,11 @@ scm->json)) (define (sources-json-builder) - "Return a JSON page listing all the sources. - -See ." + "Return a JSON page listing all the sources." + ;; The Software Heritage format is described here: + ;; https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources.json + ;; And the loader is implemented here: + ;; https://forge.softwareheritage.org/source/swh-loader-core/browse/master/swh/loader/package/nixguix/ (define (package->json package) `(,@(if (origin? (package-source package)) (origin->json (package-source package)) @@ -185,7 +200,13 @@ See ." (make-page "sources.json" `(("sources" . ,(list->vector (map package->json (all-packages)))) - ("version" . "1")) + ("version" . "1") + ("revision" . + ,(match (current-profile) + (#f %guix-version) ;for lack of a better ID + (profile + (let ((channel (find guix-channel? (profile-channels profile)))) + (channel-commit channel)))))) scm->json)) (define (index-builder) -- 2.26.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 29 13:01:24 2020 Received: (at 42019) by debbugs.gnu.org; 29 Jun 2020 17:01:24 +0000 Received: from localhost ([127.0.0.1]:49304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpx9o-0002U3-3l for submit@debbugs.gnu.org; Mon, 29 Jun 2020 13:01:24 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:39222) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpx9k-0002To-U0 for 42019@debbugs.gnu.org; Mon, 29 Jun 2020 13:01:22 -0400 Received: by mail-qk1-f196.google.com with SMTP id l6so15904303qkc.6 for <42019@debbugs.gnu.org>; Mon, 29 Jun 2020 10:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=v8ULMbuhnF4Updl+hUDJ5NTtMLG3FSVCnN+O0/rZRuQ=; b=iMJ1X2cV6J52f1Es9JVWxVdu7fgrO7opBfo9u8/3BiAB8m9qpQI+kU7SSmUHVBu37X mYRaR9hSEABLByFfMrQRtAbHbunvG25i9WsTFjunXubrttoSfeYB5Dj1zwE2//Wqp5Jz kyINwyD95b4t3wvtXZAtEQ8v1FsR6VO4uq0o8XUMDyql5HfXvl0p+25q6nBWTwpyNlf9 vB39EQCmp6tk+ptADq+418nakIEOACCUQ9znsTWCYHvFHeqRaj96Kwm7fZXJhbDfjZuY oXmBb4zPT49/71AgMEUQsMUMQBu+TgkFvylkFWm1acO54cK2lNFWaR94zxV2ULWiM10L MBzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=v8ULMbuhnF4Updl+hUDJ5NTtMLG3FSVCnN+O0/rZRuQ=; b=bv8u3ZA0HBkGiOokHaiBW2KChF58iBknav9thvEc2spsqXM6jD5Ad0MjLngKwtyw4Y tyXRNMhUa2eu7NmvKPIGTrqqvmNxxv/oNnYI3M6DJodz3h5nfkXst+HIoxOLAL8jFNxm oAdjIo+IjbFzxo4ZCnAStDPI5DsBhw48ECA2C+OUDJ2D5mtYHTpRWKkmkCsNEaU7w5rJ IiA+N2JdZ5cj1chuwXchZO8bREaix7aQBfOZJK8B+kPhy60PaEzNPt8iLZq0WaA9f3gv hvATUuxd9GYDobg2fLVczt5WltheKPuXdV6mFTCXYiWpm7q8Ezk0q/7Pb2RxF6OoGtAF 5hgA== X-Gm-Message-State: AOAM533mKzMMGLO635ie7S0S5fMyncVgWFqC3YVmzhfo8atgKnuJvx+q SJFqJVWwNKPl5kWCtjzNs4ELkhhCxXHJht2z30rWbLal X-Google-Smtp-Source: ABdhPJxj2WkerzSeIjUFfyCs0yaojz0yeh26NH/YlMrBfIoqPsvpWHQ8MWYThJM0L8MdJ0WHRakuFOcGtfr9EIL52rE= X-Received: by 2002:a05:620a:567:: with SMTP id p7mr7991293qkp.232.1593450075198; Mon, 29 Jun 2020 10:01:15 -0700 (PDT) MIME-Version: 1.0 References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> <87zh8ol9qp.fsf@gnu.org> <86y2o8be3c.fsf@gmail.com> In-Reply-To: <86y2o8be3c.fsf@gmail.com> From: zimoun Date: Mon, 29 Jun 2020 19:01:04 +0200 Message-ID: Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: 42019@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, On Sat, 27 Jun 2020 at 19:42, zimoun wrote: > Thanks for the tip, I did not know. I will sent a v2 with your > suggestion or feel free to update the patch and push it. :-) v2 is sent. BTW, in the SWH picture and after a chat video with lewo, I do not think that the website is the right place. Instead, it should go to ci.guix.gnu.org or data.guix.gnu.org. Or maybe integrated with "guix publish". Well, the next step is to have a collection of sources.json -- that the point of "revision". WDYT? Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 29 16:42:09 2020 Received: (at 42019) by debbugs.gnu.org; 29 Jun 2020 20:42:09 +0000 Received: from localhost ([127.0.0.1]:49620 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jq0bB-0003xp-W9 for submit@debbugs.gnu.org; Mon, 29 Jun 2020 16:42:09 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56226) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jq0b9-0003xa-CL for 42019@debbugs.gnu.org; Mon, 29 Jun 2020 16:41:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:38265) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jq0b3-0005sY-Sr; Mon, 29 Jun 2020 16:41:45 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=46638 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jq0b2-0004f7-Tm; Mon, 29 Jun 2020 16:41:45 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> <87zh8ol9qp.fsf@gnu.org> <86y2o8be3c.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 12 Messidor an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 29 Jun 2020 22:41:43 +0200 In-Reply-To: (zimoun's message of "Mon, 29 Jun 2020 19:01:04 +0200") Message-ID: <87v9j9a9ko.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 42019 Cc: 42019@debbugs.gnu.org, Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, zimoun skribis: > BTW, in the SWH picture and after a chat video with lewo, I do not > think that the website is the right place. Instead, it should go to > ci.guix.gnu.org or data.guix.gnu.org. Or maybe integrated with "guix > publish". Well, the next step is to have a collection of sources.json > -- that the point of "revision". WDYT? The Guix Data Service would be a natural place for =E2=80=98sources.json=E2= =80=99 IMO. Thoughts, Chris? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 29 19:28:56 2020 Received: (at 42019) by debbugs.gnu.org; 29 Jun 2020 23:28:56 +0000 Received: from localhost ([127.0.0.1]:49754 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jq3Cq-0003t6-Jo for submit@debbugs.gnu.org; Mon, 29 Jun 2020 19:28:56 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:41593) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jq3Cn-0003ss-2n for 42019@debbugs.gnu.org; Mon, 29 Jun 2020 19:28:56 -0400 Received: by mail-wr1-f68.google.com with SMTP id z15so7000734wrl.8 for <42019@debbugs.gnu.org>; Mon, 29 Jun 2020 16:28:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=F3EVIunIoa/MUt7LWRHPF1STO0NS6SMWe53N1h8Yj/Q=; b=HI8SrPkQ2GJpUnE/hvnyUPgP9dPnvzHQq25bySUmEMB+bMQDVNLy0MOOXuF4XOt+Gb 0kUae73rsN75kqKe9d7odY6bMgMCn0/02a2zwNtQMVfaah8Z51v90Ps7Jyourb640Ijw Ku1s9+c+FkoQJ6hlQV4fKplG1S9JtQmYh82V0fCJyKjL+UbgS/shlksOehEAy8Ab3Ry9 sB3dgZKDTMlRZAQlMh0qHZptt23qm7tfGZUCyi/4g+D4VtTPdu1PwrJ+TTbT7mpxsebn et/4IIJq8PVC6u1/HBJJeuBkX1Xj3/vh2e6TN+GBARaDVQ1CnEfri4tYL4LxQl+mhBjw 1bag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=F3EVIunIoa/MUt7LWRHPF1STO0NS6SMWe53N1h8Yj/Q=; b=AgZJ+FOvFlJrw4vdtGpRZEM2sQvxznyh5D6lbR86Eg7Qf0Z6Li4v7eIk0LVlrpWAXC /6K486817w66Z75npu5kZGGJBIqvSEwBL3MVdEBVX/WT0tKmIAgOjZdMp3mAKYFaj2/j NpGfJKuSHGXdJkPucGWRXrP+/yDJwRNXTHCRiohjsHo1ofu+i5Bd0vy5mox5PuY9BTIT Ph0TEGxUHL+mcc2DNddddozyfjROVJdQi8TIyztBNsJ3oER+uWGe3P/KE+4gLmwbMW+n 6Jl/82GT7L/uZ0KR09o3CYBrgL3CssFVHzYH5NKZwoWp3W5fWu0TKh3wrq6Tjs5X5yNx rFLw== X-Gm-Message-State: AOAM5332tNmzBNQFYILtI87rHiOwE3ZISB0pOMg5KaT8jCRZxDbZEM0k 6F4AhKAJ2km/5mBf8YI0gvg= X-Google-Smtp-Source: ABdhPJw3tmWBYAHUSrG/Ys1e+z8fpH47hrKSa+hXNit1sTY/jNBjbxoFG/6dZM7EEYRDX199Y33Hew== X-Received: by 2002:a5d:40cf:: with SMTP id b15mr18701563wrq.319.1593473326845; Mon, 29 Jun 2020 16:28:46 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id d13sm1420879wrn.61.2020.06.29.16.28.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jun 2020 16:28:46 -0700 (PDT) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. In-Reply-To: <87v9j9a9ko.fsf@gnu.org> References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> <87zh8ol9qp.fsf@gnu.org> <86y2o8be3c.fsf@gmail.com> <87v9j9a9ko.fsf@gnu.org> Date: Tue, 30 Jun 2020 01:28:45 +0200 Message-ID: <864kqtbgeq.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: 42019@debbugs.gnu.org, Christopher Baines X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Chris, On Mon, 29 Jun 2020 at 22:41, Ludovic Court=C3=A8s wrote: > zimoun skribis: > >> BTW, in the SWH picture and after a chat video with lewo, I do not >> think that the website is the right place. Instead, it should go to >> ci.guix.gnu.org or data.guix.gnu.org. Or maybe integrated with "guix >> publish". Well, the next step is to have a collection of sources.json >> -- that the point of "revision". WDYT? > > The Guix Data Service would be a natural place for =E2=80=98sources.json= =E2=80=99 IMO. > Thoughts, Chris? If it goes to the GDS, then first let point me where to start. :-) And second, it could be nice in the "near" future to have at least 2 sources.json: one for the last commit refreshed every X minutes (or hours) and another one containing the concatenation of all the sources of Guix (at least the one reachable by guix time-machine i.e. after the big overhaul of Inferiors). I will go on #swh-devel or reach lewo to know how "near" it is on SWH side. Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 01 15:35:32 2020 Received: (at 42019) by debbugs.gnu.org; 1 Jul 2020 19:35:32 +0000 Received: from localhost ([127.0.0.1]:53758 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqiW3-0002AH-OM for submit@debbugs.gnu.org; Wed, 01 Jul 2020 15:35:31 -0400 Received: from mira.cbaines.net ([212.71.252.8]:52008) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqiW1-0002A9-Vx for 42019@debbugs.gnu.org; Wed, 01 Jul 2020 15:35:30 -0400 Received: from localhost (unknown [46.237.173.5]) by mira.cbaines.net (Postfix) with ESMTPSA id C018F27BBE1; Wed, 1 Jul 2020 20:35:28 +0100 (BST) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 6a477919; Wed, 1 Jul 2020 19:35:26 +0000 (UTC) References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> <87zh8ol9qp.fsf@gnu.org> <86y2o8be3c.fsf@gmail.com> <87v9j9a9ko.fsf@gnu.org> <864kqtbgeq.fsf@gmail.com> User-agent: mu4e 1.4.10; emacs 26.3 From: Christopher Baines To: zimoun Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. In-reply-to: <864kqtbgeq.fsf@gmail.com> Date: Wed, 01 Jul 2020 20:35:24 +0100 Message-ID: <87d05f2flv.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 42019@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable zimoun writes: > Hi Chris, > > On Mon, 29 Jun 2020 at 22:41, Ludovic Court=C3=A8s wrote: >> zimoun skribis: >> >>> BTW, in the SWH picture and after a chat video with lewo, I do not >>> think that the website is the right place. Instead, it should go to >>> ci.guix.gnu.org or data.guix.gnu.org. Or maybe integrated with "guix >>> publish". Well, the next step is to have a collection of sources.json >>> -- that the point of "revision". WDYT? >> >> The Guix Data Service would be a natural place for =E2=80=98sources.json= =E2=80=99 IMO. >> Thoughts, Chris? > > If it goes to the GDS, then first let point me where to start. :-) I think this does sound like a good use of the Guix Data Service. Unfortunately, the sources of packages aren't currently stored in the Guix Data Service database, so I'm guessing this will require storing some new data, then working out how to present it. A question maybe for you Simon, what would be the perfect data for this particular use case? I gather it's something about the (source ...) field in packages, probably for all the exported (plus maybe not-exported packages). > And second, it could be nice in the "near" future to have at least 2 > sources.json: one for the last commit refreshed every X minutes (or > hours) and another one containing the concatenation of all the sources > of Guix (at least the one reachable by guix time-machine i.e. after the > big overhaul of Inferiors). I will go on #swh-devel or reach lewo to > know how "near" it is on SWH side. Once you can get the data for an individual revision in the Guix Data Service, it should be reasonably easy to just get the data for multiple revisions, say all for the last week. Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl785XxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XdWAg/+JwSny86BDKVjs5kSTrfUZb5tGePYgVqHZCYwDsx0AuEOdsx2IcHsuXQe wJAc4TS47LS4SzkFI3IVrNK9COYTuKoYBX5H81+g0zJTyNWKd01sMDS+4XVsPAAq B3yEmuA3z9j0STR0J7dI1vhP/nToe11ktOzY3/WlaQkbJan51ORrK16FjsDqCiwF SD/E3I1ilZcWyHWiwDzEAfPODNjZZ/UURr96qkSERbQtmuIoLCsbIzTPzC/hQgqC OgXFYwPG09LdtXYdTW9yqq16NNMt3YkywpTCVnbjs+m831C4GrmNzgFDJ3pqP5rc G/033lNXumPixWzUY48cRCxvG/2krVXamz7TmjFAhDF9rz6E055YM0nGUxVCh6tA zh2un5Wvcd/MH1N/AeuwU9F5pdSKMJRxNHXdv/udQmc+0pWNxfK0rjU3CPAMss1i uJxJ5StBklrfdaOHkT+JTaJ4h5dnNFWAaIBgu1mGfGo4Soo2AFxmH/dkcLJg6jbL Yh/aL95jqBbFleEn11HUuyiwZ3ouP48Gv9ZcwxNPdeSjiFjHDJk2bjKwv5UWrwwk mTQrWhhVTgzbYwoWRCffa7DySU9j5gfBphUSaqAPXmrWoAHPEHoN03MZJdxEpe88 IlvlHSNZUDg0GK1OxRtg+0eF2gE/yA7Fs9ZDSL705grvDAGek5g= =qnpz -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 01 16:29:10 2020 Received: (at 42019) by debbugs.gnu.org; 1 Jul 2020 20:29:10 +0000 Received: from localhost ([127.0.0.1]:53773 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqjLy-0003Ny-FZ for submit@debbugs.gnu.org; Wed, 01 Jul 2020 16:29:10 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:37879) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqjLw-0003Nk-A0 for 42019@debbugs.gnu.org; Wed, 01 Jul 2020 16:29:08 -0400 Received: by mail-wr1-f68.google.com with SMTP id a6so25483191wrm.4 for <42019@debbugs.gnu.org>; Wed, 01 Jul 2020 13:29:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=ihIXhy2RYcqgp1oLqgltC995BonpOoqnCZLpCCfXFpo=; b=YTIU78W2+BvawBnuV7v8Xcl4fAqI3o+GaESgzmhWYjA1SPNDP9nsR/aiSu0I6cBfG5 fVRxm6Y8T2+twovKwrdjllsURNbhsUKkzIeLDTQQREuBsUctkvIBU2oV48ifDcMVyVT0 hpzHtb4l+hqLaKkySrQbJjQHsNoavCgHzbDEvj9jpQWN0gF5r1+0oqDnWJUA0H3NTQIM 76MFUOur2VPmVWQ1iWiFH/iCtn1JGaH+1We/Nw7C85jFGRv3u5KvdklNTj9rhsHDh4xJ haJmMl6rUI0qJUaiE5musac8H43FQJweYPc+WD9C2EjfwnIjKfPdu00kM5GDQHXsyrhF +IOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=ihIXhy2RYcqgp1oLqgltC995BonpOoqnCZLpCCfXFpo=; b=aY1y+/8F14Bv8awXcCaB6ZZZGIO43bJLuHydqor2ZcYBgkMGYPDEn3nxpb/G/6xoSz qZ/zPs1twLFqNjY5SzRsykjk+/J0HzTLT2Ulqoq5LFBp0AK0ZcGLq61O3YqVl1HvlDmN FRQ551D/eXiTJit3AeXFlGkIPVCaxc003sJgqqdr0pOkaM1CkUE4VGmA2azUNa3AndfJ e7ryq6DQWn9wXnDGUGvw57XdcMG9m8n8jstHZliA5zrWJ6i54z1oLhUgg7SDSZ1hSqhH YnJkOuxnOhNGRv8WRNgZTkLN4Yy5mR70LyI7KvO4t7figAlwzzLaQ1U9PBGHAyukKHpr aQQg== X-Gm-Message-State: AOAM5323nsk34uKIShxS78aYY+egrSbSM1aKXWrdlHhFTd8cg7Om9TRn m4cw3GbpvV4MksTGVifUy/P1j4yyv14= X-Google-Smtp-Source: ABdhPJz0Cw4UB7HdUw2sCGgIWfeWh/I9MqYuWVHsji/yjikOPh+ZawXn4/Tsf+LC5k1+SgXz1xiQ1g== X-Received: by 2002:a5d:424f:: with SMTP id s15mr18859967wrr.342.1593635342148; Wed, 01 Jul 2020 13:29:02 -0700 (PDT) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id s10sm8576593wme.31.2020.07.01.13.29.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jul 2020 13:29:01 -0700 (PDT) From: zimoun To: Christopher Baines Subject: Re: [bug#42019] [PATCH 1/1] website: Add integrity to JSON sources. In-Reply-To: <87d05f2flv.fsf@cbaines.net> References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200623152139.512-1-zimon.toutoune@gmail.com> <87zh8ol9qp.fsf@gnu.org> <86y2o8be3c.fsf@gmail.com> <87v9j9a9ko.fsf@gnu.org> <864kqtbgeq.fsf@gmail.com> <87d05f2flv.fsf@cbaines.net> Date: Wed, 01 Jul 2020 22:29:00 +0200 Message-ID: <86366b9dyr.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42019 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 42019@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Chris, On Wed, 01 Jul 2020 at 20:35, Christopher Baines wrote: > A question maybe for you Simon, what would be the perfect data for this > particular use case? I gather it's something about the (source ...) > field in packages, probably for all the exported (plus maybe > not-exported packages). Currently the website builds source.json by using 'fold-packages' (traversing all the modules and returning all the public variables, if I read correctly) then excluding 'package-superseded' and 'package-replacement'. Well, maybe an example is simpler than a lot of words. The resulting JSON looks like: --8<---------------cut here---------------start------------->8--- { "type": "url", "urls": [ "https://ftpmirror.gnu.org/gnu/a2ps/a2ps-4.14.tar.gz", "ftp://ftp.cs.tu-berlin.de/pub/gnu/a2ps/a2ps-4.14.tar.gz", "ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/a2ps/a2ps-4.14.tar.gz", "http://ftp.gnu.org/pub/gnu/a2ps/a2ps-4.14.tar.gz" ], "integrity": "sha256-866NPUVkpBtuKiHyN9LysQT0gQhZHouDSXUAGCo6s6Q=" }, { "type": "git", "git_url": "https://github.com/opencog/agi-bio.git", "git_ref": "b5c6f3d99e8cca3798bf0cdf2c32f4bdb8098efb" }, --8<---------------cut here---------------end--------------->8--- So basically, the data are: origin-method, origin-uri (implies reference URLs and {git,hg,svn}-{commit,revision}), origin-hash (implies content-hash-{value,algorithm}). Note that the list of mirrors are necessary too. I have given a look to http://git.savannah.gnu.org/cgit/guix/data-service.git/tree/ but I am not sure to understand where the SQL table is defined. Thanks, simon From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 06 06:20:30 2020 Received: (at 42019-done) by debbugs.gnu.org; 6 Jul 2020 10:20:30 +0000 Received: from localhost ([127.0.0.1]:33651 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jsOEg-00088g-Ff for submit@debbugs.gnu.org; Mon, 06 Jul 2020 06:20:30 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43058) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jsOEe-00088U-8p for 42019-done@debbugs.gnu.org; Mon, 06 Jul 2020 06:20:29 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51639) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jsOEZ-0006j5-1y; Mon, 06 Jul 2020 06:20:23 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=50506 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jsOEY-0007uv-Hs; Mon, 06 Jul 2020 06:20:22 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Re: [bug#42019] [PATCH v2] website: Add integrity to JSON sources. References: <20200623151323.29639-1-zimon.toutoune@gmail.com> <20200629165057.9451-1-zimon.toutoune@gmail.com> Date: Mon, 06 Jul 2020 12:20:20 +0200 In-Reply-To: <20200629165057.9451-1-zimon.toutoune@gmail.com> (zimoun's message of "Mon, 29 Jun 2020 18:50:57 +0200") Message-ID: <87fta5kkrf.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 42019-done Cc: 42019-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, zimoun skribis: > * website/apps/packages/builder.scm (origin->json): Add integrity field u= sing > SRI format. I added missing bits to the commit log and pushed as 35bb77108fc7f2339da0b5be139043a5f3f21493 to guix-artwork.git. Thanks, and apologies for the delay! Ludo=E2=80=99. From unknown Tue Jun 17 21:50:28 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 03 Aug 2020 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator