From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: mail@nicolasgoaziou.fr, guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Nicolas Goaziou , X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Nicolas Goaziou , Received: via spool by submit@debbugs.gnu.org id=B.16463420188387 (code B ref -1); Thu, 03 Mar 2022 21:14:01 +0000 Received: (at submit) by debbugs.gnu.org; 3 Mar 2022 21:13:38 +0000 Received: from localhost ([127.0.0.1]:43923 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPslW-0002BD-EZ for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:13:38 -0500 Received: from lists.gnu.org ([209.51.188.17]:56718) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPslT-0002B4-Vl for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:13:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34888) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPslT-0000DC-PJ for guix-patches@gnu.org; Thu, 03 Mar 2022 16:13:35 -0500 Received: from [2001:470:142:3::e] (port=58802 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPslT-0006ny-Co; Thu, 03 Mar 2022 16:13:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=ZhT0X9ZeOlFlFSMn0oLBwuMcsNNYusHsyznJj450XNw=; b=r2MQLtOj9yNVA1 bD3y613EqoKHzjDBJQG15P+sFLDWDPReUuAJiH8eMYxhA9ektw+/tpR8AIA9Y/d9CkYSmtj91Yl92 eQlwNd6Upn2xPCyMSPyt1tG+8/qdrnsQExqAU3QZ3gjC731gXGkgRQgRs4tHt7HKGMfxe9h4wcuas fdQkdRJ+712RZJVuycN8+r8RsMFx1+m/mCqu/wGYSHNmzMpC26uRr0Z0lD/1be6lExckYZ9iU8hDV n0L/RymWGqKJkwHg9QssKMLvLW9LN6ozXwpMBBoqVXlT/chSUL7X4JVGlh3l28rLBRBwpoZ3N99bt IEx3vnLv8ih20zB/yNtg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50078 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPslS-0006DC-Ut; Thu, 03 Mar 2022 16:13:35 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:13:26 +0100 Message-Id: <20220303211326.19884-1-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Guix! These patches address a famous complaint about “the GitHub problem” that affects ‘guix refresh’¹, shown here in its naked awfulness: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix refresh gnu/packages/zig.scm:32:13: zig would be upgraded from 0.9.0 to 0.9.1 [...] In guix/scripts/refresh.scm: 578:14 5 (_ _) In srfi/srfi-1.scm: 634:9 4 (for-each # _) In guix/scripts/refresh.scm: 378:2 3 (check-for-package-update # (#<) …) In guix/import/github.scm: 232:12 2 (latest-release _) In ice-9/boot-9.scm: 1685:16 1 (raise-exception _ #:continuable? _) 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: Error downloading release information through the GitHub API. This may be fixed by using an access token and setting the environment variable GUIX_GITHUB_TOKEN, for instance one procured from https://github.com/settings/tokens --8<---------------cut here---------------end--------------->8--- With this change, ‘guix refresh’ warns you when the GitHub rate limit is reached, but it keeps going, falling back to the ‘generic-git’ updater if it’s among the applicable updaters: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix refresh -t github,generic-git [...] guix refresh: warning: GitHub rate limit exceeded; disallowing requests for 1477 seconds hint: You can waive the rate limit by setting the `GUIX_GITHUB_TOKEN' environment variable to a token obtained from `https://github.com/settings/tokens' with your GitHub account. Alternatively, you can wait until your rate limit is reset, or use the `generic-git' updater instead. gnu/packages/zile.scm:113:14: warning: no tags were found for zile-on-guile gnu/packages/zig.scm:32:13: zig would be upgraded from 0.9.0 to 0.9.1 gnu/packages/xorg.scm:2830:7: warning: no valid tags found for xf86-video-freedreno gnu/packages/xml.scm:2132:13: java-kxml2 would be upgraded from 2.4.2 to 2.5.0 --8<---------------cut here---------------end--------------->8--- The GitHub updater becomes functional again once the rate limit has been reset. The code to deal with rate limiting is similar to that in (guix swh). Thoughts? Thanks, Ludo’. ¹ https://issues.guix.gnu.org/53818#50 Ludovic Courtès (4): http-client: Add response headers to '&http-get-error'. import: github: Gracefully handle rate limit exhaustion. http-client: Correctly handle redirects when #:keep-alive? #t. import: github: Reuse HTTP connection for the /tags URL fallback. .dir-locals.el | 1 + guix/http-client.scm | 39 +++++++++----- guix/import/github.scm | 119 +++++++++++++++++++++++++++++------------ 3 files changed, 112 insertions(+), 47 deletions(-) base-commit: be84fb701bf7a36a0eb50147ccbb988aa3f41209 -- 2.34.0 From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 1/4] http-client: Add response headers to '&http-get-error'. References: <20220303211326.19884-1-ludo@gnu.org> In-Reply-To: <20220303211326.19884-1-ludo@gnu.org> Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421258666 (code B ref 54241); Thu, 03 Mar 2022 21:16:02 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:25 +0000 Received: from localhost ([127.0.0.1]:43939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnE-0002Fd-TF for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:25 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57758) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnC-0002FM-RT for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:24 -0500 Received: from [2001:470:142:3::e] (port=58918 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsn7-00078m-LD; Thu, 03 Mar 2022 16:15:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=KbRkngTko0T7lcr8FWw0zRyzmJMdgMzzPqVooLXmPlE=; b=YDFscPTLrHTpBE 9GNIhs2cpaE0s2coo40j0IlBRQtLyFSPtFl0kuihbGYOYp55Ra03VuGpNSKN2zqgv195D+7qBTdCr Jsb5PffUBKHPthZ8fj2eyBIGCUgh/D/Ag9LloCqGk2WtSfmaHzuZPb603sZ9Fey8dq7rN3pZ0B+D9 JXiva/E4vFJX+J5/+6YF54+WFgwG2Cj99xh2nZu2k94zabYwDhNYHAoHVJPdgARGgbDIz/pFWuzBs XD5nnP9VTREpUiHbWGwS8/6fYrTD6EKqYfkcSp8nYPAUAwCFWR1lQCtAW3bOk2YzLZeN5/QeKsXFQ lv3bI2DWFq0g7o/LZhzg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsmu-0006KA-Tf; Thu, 03 Mar 2022 16:15:16 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:41 +0100 Message-Id: <20220303211444.19928-1-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/http-client.scm (&http-get-error)[headers]: New field. (http-fetch): Initialize the 'headers' field. --- guix/http-client.scm | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 10bc278023..4b01e31165 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2020, 2021 Ludovic Courtès +;;; Copyright © 2012-2018, 2020-2022 Ludovic Courtès ;;; Copyright © 2015 Mark H Weaver ;;; Copyright © 2012, 2015 Free Software Foundation, Inc. ;;; Copyright © 2017 Tobias Geerinckx-Rice @@ -52,6 +52,7 @@ (define-module (guix http-client) http-get-error-uri http-get-error-code http-get-error-reason + http-get-error-headers http-fetch http-multiple-get @@ -69,9 +70,10 @@ (define-module (guix http-client) ;; HTTP GET error. (define-condition-type &http-get-error &error http-get-error? - (uri http-get-error-uri) ; URI - (code http-get-error-code) ; integer - (reason http-get-error-reason)) ; string + (uri http-get-error-uri) ;URI + (code http-get-error-code) ;integer + (reason http-get-error-reason) ;string + (headers http-get-error-headers)) ;alist (define* (http-fetch uri #:key port (text? #f) (buffered? #t) @@ -138,7 +140,8 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) (raise (condition (&http-get-error (uri uri) (code code) - (reason (response-reason-phrase resp))) + (reason (response-reason-phrase resp)) + (headers (response-headers resp))) (&message (message (format -- 2.34.0 From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421298690 (code B ref 54241); Thu, 03 Mar 2022 21:16:02 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:29 +0000 Received: from localhost ([127.0.0.1]:43945 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnI-0002Fz-8D for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57766) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnE-0002FO-A5 for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:24 -0500 Received: from [2001:470:142:3::e] (port=58924 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsn9-000794-4S; Thu, 03 Mar 2022 16:15:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=YqTGlDNK8xosVwE96k92CcKvx/luyf2dhcbIRo7hYDU=; b=kTlPDjCbdWhaHgcd0bVk 4X+I2zmI1xyDhCPXdXO9/Y1jbGAcHiowkZFl8IdePdKhv+tV4w3KNMw6gnAc7uRxudcS+nZQeR0WY yBJ3eqWHhJyKOgtakw5XZHfjbXFl3jIDOipvishe7y4/z8OGDe4MbFPGB8AnENZ3i4K8msxapir5d cQsgWE3sPKKe0/0F4hOBUREf0WfJfJokF13yUOf12qNZiF+B+mIfLxTCtdp7BbSGBhVGqG0uXZuD8 AhsePOkOpEN/9wHZvBNKdADCvEfGEGDo2a4OSpygVHcn8YnMrPdOaGrEzxoovFNajfmX2PRcqLHn4 Z3+pSyiCw5lNFQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsn7-0006KA-Q5; Thu, 03 Mar 2022 16:15:18 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:42 +0100 Message-Id: <20220303211444.19928-2-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220303211444.19928-1-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Previously, 'guix refresh' would literally crash when the rate limit was reached due to the call to 'error'. With this change, the updater notices when the rate limit is reached and it turns itself into a no-op until the rate limit has been reset. When running "guix refresh" (with no arguments), the 'github' updater gets used until the rate limit has been reached, after which "guix refresh" automatically picks up the next valid updater, typically 'generic-git'. * guix/import/github.scm (fetch-releases-or-tags): Use 'http-fetch' directly instead of 'json-fetch' to let 'http-get-error?' exceptions through. Handle 403 errors with an 'X-RateLimit-Remaining' header. (%rate-limit-reset-time): New variable. (update-rate-limit-reset-time!, request-rate-limit-reached?): New procedures. (latest-released-version): Remove calls to 'error'. --- guix/import/github.scm | 113 +++++++++++++++++++++++++++++------------ 1 file changed, 80 insertions(+), 33 deletions(-) diff --git a/guix/import/github.scm b/guix/import/github.scm index 8c1898c0c5..5062bad80d 100644 --- a/guix/import/github.scm +++ b/guix/import/github.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2016 Ben Woodcroft -;;; Copyright © 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2017-2020, 2022 Ludovic Courtès ;;; Copyright © 2018 Eric Bavier ;;; Copyright © 2019 Arun Isaac ;;; Copyright © 2019 Efraim Flashner @@ -30,15 +30,16 @@ (define-module (guix import github) #:use-module (guix utils) #:use-module (guix i18n) #:use-module (guix diagnostics) + #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix download) #:prefix download:) #:use-module ((guix git-download) #:prefix download:) #:use-module (guix import utils) - #:use-module (guix import json) #:use-module (json) #:use-module (guix packages) #:use-module (guix upstream) #:use-module (guix http-client) #:use-module (web uri) + #:use-module (web response) #:export (%github-api %github-updater)) ;; For tests. @@ -140,6 +141,30 @@ (define %github-token ;; limit, or #f. (make-parameter (getenv "GUIX_GITHUB_TOKEN"))) +(define %rate-limit-reset-time + ;; Time (seconds since the Epoch, UTC) when the rate limit for GitHub + ;; requests will be reset, or #f if the rate limit hasn't been reached. + #f) + +(define (update-rate-limit-reset-time! headers) + "Update the rate limit reset time based on HEADERS, the HTTP response +headers." + (match (assq-ref headers 'x-ratelimit-reset) + ((= string->number (? number? reset)) + (set! %rate-limit-reset-time reset) + reset) + (_ + 0))) + +(define (request-rate-limit-reached?) + "Return true if the rate limit has been reached." + (and %rate-limit-reset-time + (match (< (car (gettimeofday)) %rate-limit-reset-time) + (#t #t) + (#f + (set! %rate-limit-reset-time #f) + #f)))) + (define (fetch-releases-or-tags url) "Fetch the list of \"releases\" or, if it's empty, the list of tags for the repository at URL. Return the corresponding JSON dictionaries (alists), @@ -170,20 +195,49 @@ (define headers `((Authorization . ,(string-append "token " (%github-token)))) '()))) - (guard (c ((and (http-get-error? c) - (= 404 (http-get-error-code c))) - (warning (G_ "~a is unreachable (~a)~%") - release-url (http-get-error-code c)) - '#())) ;return an empty release set - (let* ((port (http-fetch release-url #:headers headers)) - (result (json->scm port))) - (close-port port) - (match result - (#() - ;; We got the empty list, presumably because the user didn't use GitHub's - ;; "release" mechanism, but hopefully they did use Git tags. - (json-fetch tag-url #:headers headers)) - (x x))))) + (and (not (request-rate-limit-reached?)) + (guard (c ((and (http-get-error? c) + (= 404 (http-get-error-code c))) + (warning (G_ "~a is unreachable (~a)~%") + (uri->string (http-get-error-uri c)) + (http-get-error-code c)) + '#()) ;return an empty release set + ((and (http-get-error? c) + (= 403 (http-get-error-code c))) + ;; See + ;; . + (match (assq-ref (http-get-error-headers c) + 'x-ratelimit-remaining) + (#f + (raise c)) + ((? (compose zero? string->number)) + (let ((reset (update-rate-limit-reset-time! + (http-get-error-headers c)))) + (warning (G_ "GitHub rate limit exceeded; \ +disallowing requests for ~a seconds~%") + (- reset (car (gettimeofday)))) + (display-hint (G_ "You can waive the rate limit by +setting the @env{GUIX_GITHUB_TOKEN} environment variable to a token obtained +from @url{https://github.com/settings/tokens} with your GitHub account. + +Alternatively, you can wait until your rate limit is reset, or use the +@code{generic-git} updater instead.")) + #f)) ;bail out + (_ + (raise c))))) + + (let* ((port (http-fetch release-url #:headers headers)) + (result (json->scm port))) + (close-port port) + (match result + (#() + ;; We got the empty list, presumably because the user didn't use GitHub's + ;; "release" mechanism, but hopefully they did use Git tags. + (let* ((port (http-fetch tag-url #:headers headers)) + (json (json->scm port))) + (close-port port) + json)) + (x x)))))) (define (latest-released-version url package-name) "Return the newest released version and its tag given a string URL like @@ -223,23 +277,16 @@ (define (release->version release) (cons tag tag)) (else #f)))) - (let* ((json (and=> (fetch-releases-or-tags url) - vector->list))) - (if (eq? json #f) - (if (%github-token) - (error "Error downloading release information through the GitHub -API when using a GitHub token") - (error "Error downloading release information through the GitHub -API. This may be fixed by using an access token and setting the environment -variable GUIX_GITHUB_TOKEN, for instance one procured from -https://github.com/settings/tokens")) - (match (sort (filter-map release->version - (match (remove pre-release? json) - (() json) ; keep everything - (releases releases))) - (lambda (x y) (version>? (car x) (car y)))) - (((latest-version . tag) . _) (values latest-version tag)) - (() (values #f #f)))))) + (match (and=> (fetch-releases-or-tags url) vector->list) + (#f (values #f #f)) + (json + (match (sort (filter-map release->version + (match (remove pre-release? json) + (() json) ; keep everything + (releases releases))) + (lambda (x y) (version>? (car x) (car y)))) + (((latest-version . tag) . _) (values latest-version tag)) + (() (values #f #f)))))) (define (latest-release pkg) "Return an for the latest release of PKG." -- 2.34.0 From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 3/4] http-client: Correctly handle redirects when #:keep-alive? #t. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421298696 (code B ref 54241); Thu, 03 Mar 2022 21:16:03 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:29 +0000 Received: from localhost ([127.0.0.1]:43947 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnJ-0002G7-4z for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57770) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnF-0002FQ-Us for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:26 -0500 Received: from [2001:470:142:3::e] (port=58926 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnA-00079I-Oz; Thu, 03 Mar 2022 16:15:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=3FOgxt6RAntKXgIo3ctekWMUYLNuoI7nrkNNHJj7ILo=; b=UETos/z1a/NobplX+7K7 k/BWR4KDemZttuqwBGsmlqIyX2+2VXhsHdU/Swn5I2TJvlO+4RKkXSgoj0Krrmdd2QmVlyvU2IgUf slTxcRUqyPR935THz3526G7drgxlIoxpfPa4nDPpLxzseC9gAIzIhXCRJIuKbAeYbheXOPbmUud+o buPuCtyZU84uPtMWC1uWhgFVguB3JAITR21K/IYi4o0OnD3DeoYR7ktDYNAKlHorQqrajHM/7kHOE G2QNTWDuxZzVNongb47zeW6hnDJ8IqPFqOjF4kNA5asxogbbWu0LFj2HaNg4ew1K7AVVpuqrEtKyU +CXXViqfP/aDcQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsn9-0006KA-9e; Thu, 03 Mar 2022 16:15:20 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:43 +0100 Message-Id: <20220303211444.19928-3-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220303211444.19928-1-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Previously PORT would be closed unconditionally, which broke redirects when #:keep-alive? #t is given. * guix/http-client.scm (http-fetch): Make 'port' a parameter of 'loop'. Upon 3xx responses, do not close PORT is KEEP-ALIVE? is true, but consume RESP's body. Add second argument to 'loop'. --- guix/http-client.scm | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/guix/http-client.scm b/guix/http-client.scm index 4b01e31165..4784497e5f 100644 --- a/guix/http-client.scm +++ b/guix/http-client.scm @@ -102,12 +102,12 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) Raise an '&http-get-error' condition if downloading fails." (let loop ((uri (if (string? uri) (string->uri uri) - uri))) - (let ((port (or port (open-connection uri - #:verify-certificate? - verify-certificate? - #:timeout timeout))) - (headers (match (uri-userinfo uri) + uri)) + (port (or port (open-connection uri + #:verify-certificate? + verify-certificate? + #:timeout timeout)))) + (let ((headers (match (uri-userinfo uri) ((? string? str) (cons (cons 'Authorization (string-append "Basic " @@ -131,11 +131,19 @@ (define* (http-fetch uri #:key port (text? #f) (buffered? #t) 303 ; see other 307 ; temporary redirection 308) ; permanent redirection - (let ((uri (resolve-uri-reference (response-location resp) uri))) - (close-port port) + (let ((host (uri-host uri)) + (uri (resolve-uri-reference (response-location resp) uri))) + (if keep-alive? + (dump-port data (%make-void-port "w0") + (response-content-length resp)) + (close-port port)) (format log-port (G_ "following redirection to `~a'...~%") (uri->string uri)) - (loop uri))) + (loop uri + (and keep-alive? + (or (not (uri-host uri)) + (string=? host (uri-host uri))) + port)))) (else (raise (condition (&http-get-error (uri uri) -- 2.34.0 From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 4/4] import: github: Reuse HTTP connection for the /tags URL fallback. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 03 Mar 2022 21:16:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 54241@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16463421308704 (code B ref 54241); Thu, 03 Mar 2022 21:16:03 +0000 Received: (at 54241) by debbugs.gnu.org; 3 Mar 2022 21:15:30 +0000 Received: from localhost ([127.0.0.1]:43949 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnK-0002GK-Eh for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:30 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57782) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPsnI-0002FV-63 for 54241@debbugs.gnu.org; Thu, 03 Mar 2022 16:15:28 -0500 Received: from [2001:470:142:3::e] (port=58928 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPsnC-00079S-6p; Thu, 03 Mar 2022 16:15:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=yWJeafeePwydXKt0CTt4A2Wg0EUsJ40BZd0C7uxqd5U=; b=T8qUP8WETPgn+gygoMB8 5P6dtkg/38+jCHrH7s7bW/gSfWXrgJaB9KRkrJuJP3u8EFTbe7RDXZ95giYUiGqnxDm0ck8UfcOt3 48FuXDDBnm25U0GMkZi7QBIINVmpimcLwtbXGeDsxsee9cO5THEi5k5DT617cFjthFddH6JRVwYtY dzXcqeBcINglirLl97iunJzi9/AT2Kq0V0eFa1Qzu4JLELsaSyb8NazbjpuqZYl01Ke7j1unbHfRY uWK153XYXk0g8tEpfB//A83gxMSc/jpomRMYz67oKHGtmAE25TvBOZbPUF1QapTEJ79smqK9Aj64d rR+iDm/QQ+0ghw==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50082 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPsnA-0006KA-Un; Thu, 03 Mar 2022 16:15:21 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 3 Mar 2022 22:14:44 +0100 Message-Id: <20220303211444.19928-4-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20220303211444.19928-1-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) * guix/import/github.scm (fetch-releases-or-tags): Call 'open-connection-for-uri' and reuse the same connection for the two 'http-fetch' calls. * .dir-locals.el (scheme-mode): Add 'call-with-port'. --- .dir-locals.el | 1 + guix/import/github.scm | 30 ++++++++++++++++++------------ 2 files changed, 19 insertions(+), 12 deletions(-) diff --git a/.dir-locals.el b/.dir-locals.el index 0edf2a8d23..bee745cc27 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -52,6 +52,7 @@ (eval . (put 'test-equal 'scheme-indent-function 1)) (eval . (put 'test-eq 'scheme-indent-function 1)) (eval . (put 'call-with-input-string 'scheme-indent-function 1)) + (eval . (put 'call-with-port 'scheme-indent-function 1)) (eval . (put 'guard 'scheme-indent-function 1)) (eval . (put 'lambda* 'scheme-indent-function 1)) (eval . (put 'substitute* 'scheme-indent-function 1)) diff --git a/guix/import/github.scm b/guix/import/github.scm index 5062bad80d..84f70eed0f 100644 --- a/guix/import/github.scm +++ b/guix/import/github.scm @@ -33,6 +33,7 @@ (define-module (guix import github) #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix download) #:prefix download:) #:use-module ((guix git-download) #:prefix download:) + #:autoload (guix build download) (open-connection-for-uri) #:use-module (guix import utils) #:use-module (json) #:use-module (guix packages) @@ -226,18 +227,23 @@ (define headers (_ (raise c))))) - (let* ((port (http-fetch release-url #:headers headers)) - (result (json->scm port))) - (close-port port) - (match result - (#() - ;; We got the empty list, presumably because the user didn't use GitHub's - ;; "release" mechanism, but hopefully they did use Git tags. - (let* ((port (http-fetch tag-url #:headers headers)) - (json (json->scm port))) - (close-port port) - json)) - (x x)))))) + (let ((release-uri (string->uri release-url))) + (call-with-port (open-connection-for-uri release-uri) + (lambda (connection) + (let* ((result (json->scm + (http-fetch release-uri + #:port connection + #:keep-alive? #t + #:headers headers)))) + (match result + (#() + ;; We got the empty list, presumably because the user didn't use GitHub's + ;; "release" mechanism, but hopefully they did use Git tags. + (json->scm (http-fetch tag-url + #:port connection + #:keep-alive? #t + #:headers headers))) + (x x))))))))) (define (latest-released-version url package-name) "Return the newest released version and its tag given a string URL like -- 2.34.0 From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 04 Mar 2022 12:08:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Cc: Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.164639567514039 (code B ref 54241); Fri, 04 Mar 2022 12:08:02 +0000 Received: (at 54241) by debbugs.gnu.org; 4 Mar 2022 12:07:55 +0000 Received: from localhost ([127.0.0.1]:44655 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQ6ix-0003eM-FP for submit@debbugs.gnu.org; Fri, 04 Mar 2022 07:07:55 -0500 Received: from xavier.telenet-ops.be ([195.130.132.52]:41622) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQ6iv-0003eD-FY for 54241@debbugs.gnu.org; Fri, 04 Mar 2022 07:07:54 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by xavier.telenet-ops.be with bizsmtp id 2C7r2700K4UW6Th01C7rSl; Fri, 04 Mar 2022 13:07:51 +0100 Message-ID: <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> From: Maxime Devos Date: Fri, 04 Mar 2022 13:07:39 +0100 In-Reply-To: <20220303211326.19884-1-ludo@gnu.org> References: <20220303211326.19884-1-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-nJFe1EbcV5kjQG8bQyXG" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646395671; bh=gGgL70szgBylvsRFAPvrmr8whfBUW8wbr8g3Ch/IlRk=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=KGtPKyobKAVFJ5jXayqaOxZ+hqa06rT+m5tpvKW2owxjG2mGUE7lve8m0kNqjtQcd 8txDszpR7aQeOaPZJy27WzaGvsAKeJ2qlv8lm7WutS3xi4YTigtzBPA3eGioyd8dd7 4BgF3xu9xJPhXjyEOWg2qDqgk5JhVwrrsXKu9rK8tvTEd/zn928PPAAtrTrMNXyjxc Cnus7BoprACVL5xd5HzrqzM6NLRBi02RxC7ODKKPsh3HKfLSf+Y1k3rTAAoFNT9CUg k6cFQgecLxuBxEv9dnFti/WW/5k1hphyIizOiPLUggEY8DqgeRVnVERzOCmVYsCQRF sQLBz9z+p0nYA== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-nJFe1EbcV5kjQG8bQyXG Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:13 [+0100]: > With this change, =E2=80=98guix refresh=E2=80=99 warns you when the GitHu= b rate limit > is reached, but it keeps going, falling back to the =E2=80=98generic-git= =E2=80=99 > updater if it=E2=80=99s among the applicable updaters: > [...] WDYT of avoiding the rate limit by caching, using 'http-fetch/cached'? GitHub does not count requests setting If-Modified-Since against the rate limit (assuming the answer hasn't changed). Greetings, Maxime. --=-nJFe1EbcV5kjQG8bQyXG Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiIBCxccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7t5yAQC+3yNlyI6OSvD+d9Nk6/bCLR4U re/eXuU1WtqQYUQTbgD/ciWDY+7ouehsvsUS2Nm4XoZXgoC4yDagQtzuibXj9AE= =SckA -----END PGP SIGNATURE----- --=-nJFe1EbcV5kjQG8bQyXG-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 3/4] http-client: Correctly handle redirects when #:keep-alive? #t. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 04 Mar 2022 12:40:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.164639756325392 (code B ref 54241); Fri, 04 Mar 2022 12:40:01 +0000 Received: (at 54241) by debbugs.gnu.org; 4 Mar 2022 12:39:23 +0000 Received: from localhost ([127.0.0.1]:44677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQ7DP-0006bU-Fx for submit@debbugs.gnu.org; Fri, 04 Mar 2022 07:39:23 -0500 Received: from xavier.telenet-ops.be ([195.130.132.52]:56250) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQ7DN-0006bJ-BP for 54241@debbugs.gnu.org; Fri, 04 Mar 2022 07:39:22 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by xavier.telenet-ops.be with bizsmtp id 2CfA270094UW6Th01CfA1d; Fri, 04 Mar 2022 13:39:19 +0100 Message-ID: <69a48935f3f03ad72ac49eb7b529cf10eb7db96e.camel@telenet.be> From: Maxime Devos Date: Fri, 04 Mar 2022 13:39:05 +0100 In-Reply-To: <20220303211444.19928-3-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-3-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-eU9h7FhXBe8YEKs8H91w" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646397560; bh=uA2F0CIvhKi/Y46t0gvwl3NIoqzhptnL5nl9VJcblUM=; h=Subject:From:To:Date:In-Reply-To:References; b=SazC4xQxCfG4tF5gF9h4/OYqYLPQ5DdgJoMM1kKqp8TsbzEvEoF3r/lv7cpQ1mZEN XHSA7BswHGcMLl6Qp5Nv9bdC9pO/Ur4iihRsEAfFRtJ++2OYxo6mzRaLOEkaCmospT XD8n/2HvuC7XRCLADkBQ1KHSfK/mKsE2+eyTI3rpjdQVbFy1aXt0i01CkqXiFojF19 6anAp4hRudQHT9M1GTwzOi63TzV8HAj3Wb7ZFiQSEFP4twG6Wo2GlJWEoKN0Co73S2 qBhdOaLTX7j6flvFTEc0w+pNY8WMlhcQeIFQbtPrcBa5B+ZbE+y8ZdULKgNFfq+I2C 88z6ekleUCvxg== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-eU9h7FhXBe8YEKs8H91w Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: > Previously PORT would be closed unconditionally, which broke redirects > when #:keep-alive? #t is given. >=20 > * guix/http-client.scm (http-fetch): Make 'port' a parameter of 'loop'. > Upon 3xx responses, do not close PORT is KEEP-ALIVE? is true, but consume > RESP's body.=C2=A0 Add second argument to 'loop'. > --- HTTP things can become complicated, with lots of edge cases. Could an appropriate test be added to prevent this becoming buggy in the future, in case of future changed to 'http-fetch'? And a few source code comments about what is going on exactly? Greetings, Maxime. --=-eU9h7FhXBe8YEKs8H91w Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiIIaRccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7mY4AP9b1RFl4t/PnFRNRi+y/HfBqtKn EXsxXoUozot4vTcXLAD/SotzdhfuyoAdJjYmV+BGZek8ka8JBFQhtmclYdBzWwo= =fH/p -----END PGP SIGNATURE----- --=-eU9h7FhXBe8YEKs8H91w-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 04 Mar 2022 20:46:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org, Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.164642675018722 (code B ref 54241); Fri, 04 Mar 2022 20:46:01 +0000 Received: (at 54241) by debbugs.gnu.org; 4 Mar 2022 20:45:50 +0000 Received: from localhost ([127.0.0.1]:46900 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQEoA-0004ru-0J for submit@debbugs.gnu.org; Fri, 04 Mar 2022 15:45:50 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40004) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQEo7-0004rh-Nr for 54241@debbugs.gnu.org; Fri, 04 Mar 2022 15:45:48 -0500 Received: from [2001:470:142:3::e] (port=58274 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQEo0-0001c0-Cz; Fri, 04 Mar 2022 15:45:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=vm05TFwbWBvGuPqlFh3QwAB81KPXcWWAKLLUWLTKCCA=; b=LlCxPB5POe/QWVRRYz1W mAI14kx4S8kPYcJgrU+8aYyrRsvvALNQRZ4E+w3rZpdyvVKWA+3TSfSG/LHi3QmjCJYlDG0ASydTh VERX/GsT+TDPL7273e1X1F4QxuWigoExsnUqZx/SkeWtNvkr+D+SMbjGHPf1n42/DS5VqyC6902pl K883OynM1sWH1B6s6DT5O/GNVtYRQLqhCsrcBxhCHv3qkOTDEGgd+kX047ee6+1ZbE9uInTGgxWvd 2YakX35+1C3ulOjw1NN/Fq9StEm897cNf3oU4fhMkZzNkg2wtJCi9te8QY+YmQkseSf1wzHkZZZum Cx4j13n1gOA/7g==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50110 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQEnz-0002XD-DG; Fri, 04 Mar 2022 15:45:40 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> Date: Fri, 04 Mar 2022 21:45:36 +0100 In-Reply-To: <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> (Maxime Devos's message of "Fri, 04 Mar 2022 13:07:39 +0100") Message-ID: <87tucdqwen.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:13 [+0100]: >> With this change, =E2=80=98guix refresh=E2=80=99 warns you when the GitH= ub rate limit >> is reached, but it keeps going, falling back to the =E2=80=98generic-git= =E2=80=99 >> updater if it=E2=80=99s among the applicable updaters: >> [...] > > WDYT of avoiding the rate limit by caching, using 'http-fetch/cached'? > GitHub does not count requests setting If-Modified-Since against the > rate limit (assuming the answer hasn't changed). My concern is that we=E2=80=99d end up caching one or two little files in ~/.cache for each candidate package, and (rate limit aside) the overhead of dealing with the cache might outweigh the benefits. I=E2=80=99d rather = use =E2=80=98http-fetch/cached=E2=80=99 for bigger files, like in (guix cve). WDYT? My goal here was to ensure the =E2=80=98github=E2=80=99 updater doesn=E2=80= =99t get in the way of those who don=E2=80=99t want to specify a token. Thanks, Ludo=E2=80=99. From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 09:36:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16464729275271 (code B ref 54241); Sat, 05 Mar 2022 09:36:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:35:27 +0000 Received: from localhost ([127.0.0.1]:47285 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQow-0001Mw-RJ for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:35:27 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:51586) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQot-0001Mm-0l for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:35:25 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by laurent.telenet-ops.be with bizsmtp id 2ZbL2700i4UW6Th01ZbLAE; Sat, 05 Mar 2022 10:35:21 +0100 Message-ID: <81f212aeddb345789105c8b8b90bce032adb6c1a.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 10:35:15 +0100 In-Reply-To: <20220303211444.19928-2-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-w3cNZagezVWOaxzF2RiQ" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646472921; bh=wh4HXBbxWyD5hbwM6tVZwXS0z3RE/7ke1iY8w0p1xdw=; h=Subject:From:To:Date:In-Reply-To:References; b=fLvz7IdmrPmAttLtaiMKGNHUq41N9yB+tMUxUykUVyTyPtMFVlrHKjgWTVnpmdZtP yP7TSX3fbB8Bs+hAr4+8wILZG/5TEG9uwLlQKpgGA8QxYw0QgLSWui8HAk8TP5d95f 8WaTjhG1Nd27N1BrEtziJg+7BmfnjqCUa5iEgUIUo1O8eYj9Klbq9yI7OaXleNVLoI 1Co/+rYc7nmuiJvRTMke63RYsZ8qxm4oG98KkLZ6EuKGH98eDppY/pn7Sm4Wzb0X6m NpKIMwbr1yGM/zGaFOisj1+w1pJEJUYFgjA2aYbtSux+wsBKVJ1Kt9RsMANhQYIITQ a0l2oon8H8vyQ== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-w3cNZagezVWOaxzF2RiQ Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: > When running "guix refresh" (with no arguments), the 'github' updater > gets used until the rate limit has been reached, after which "guix > refresh" automatically picks up the next valid updater, typically > 'generic-git'. Wouldn't this make "guix refresh" non-deterministic (though this might be an acceptable trade-off)? Greetings, Maxime. --=-w3cNZagezVWOaxzF2RiQ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMu0xccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rXRAPwLumHwTqOkNKP+sZJTFaAEP9Oc jSM+ZTWfocFIVNXcLQD8DcVpUQEGajClDrGKwkQ/0PiFUxpef1j7lvyA6O6V9g4= =VSK6 -----END PGP SIGNATURE----- --=-w3cNZagezVWOaxzF2RiQ-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 09:38:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16464730255457 (code B ref 54241); Sat, 05 Mar 2022 09:38:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:37:05 +0000 Received: from localhost ([127.0.0.1]:47294 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQqX-0001Px-Fi for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:37:05 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:54148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQqW-0001Po-3a for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:37:04 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by laurent.telenet-ops.be with bizsmtp id 2Zd32700C4UW6Th01Zd3Pj; Sat, 05 Mar 2022 10:37:03 +0100 Message-ID: From: Maxime Devos Date: Sat, 05 Mar 2022 10:37:03 +0100 In-Reply-To: <20220303211444.19928-2-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-PcJijoVb93THoeldWDgw" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646473023; bh=B1LGmFk3RcwhZFzigOn/HLSeyekBQL/6l7u5N9moCVo=; h=Subject:From:To:Date:In-Reply-To:References; b=MjusIsQoXP3RcWOoWtoOobpn70RfDlFOotBZnHjTJp+SDLxzr2Tn8ix1sWXDWcilB D+eGwDkk0TLniFW8Mr5XSOIKXvPTPVwu2RZWzsXj34/69VdjuNbGkLvaj5WpwqZJKz SflBZ9PQrSvFC17S+PQ6FjdN7wtKNzALiWTQI/76h6bgYbRlSoiFMLD7wLmLo38+gt aYeK9vGJvbGFpABqTwDIEDXXYov795ccZoYuTKUmpFQ7quLDxMquM4cfNUZWhWU5YW /qtuiIa5YY0fRk74VbXmizyuLZ5aGFXtYne27e+mJo8jCzm8GO4uZvnpHRdzFfHIda 1YOeZIJV/U2MA== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-PcJijoVb93THoeldWDgw Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 THVkb3ZpYyBDb3VydMOocyBzY2hyZWVmIG9wIGRvIDAzLTAzLTIwMjIgb20gMjI6MTQgWyswMTAw XToKPiArwqAgKGFuZCAobm90IChyZXF1ZXN0LXJhdGUtbGltaXQtcmVhY2hlZD8pKQo+ICvCoMKg wqDCoMKgwqAgKGd1YXJkIChjIChbLi4uXQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgIDs7IFNlZQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDs7IDxo dHRwczovL2RvY3MuZ2l0aHViLmNvbS9lbi9yZXN0L292ZXJ2aWV3L3Jlc291cmNlcy1pbi10aGUt cmVzdC1hcGkjcmF0ZS1saW1pdGluZz4uCj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqAgKG1hdGNoIChhc3NxLXJlZiAoaHR0cC1nZXQtZXJyb3ItaGVhZGVycyBjKQo+ICvCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCAneC1yYXRlbGltaXQtcmVtYWluaW5nKQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCAoI2YKPiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCAocmFpc2UgYykpCj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgICgoPyAoY29tcG9zZSB6ZXJvPyBzdHJpbmctPm51bWJlcikpCj4gK8KgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgKGxldCAoKHJlc2V0ICh1cGRhdGUtcmF0ZS1saW1p dC1yZXNldC10aW1lIQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAoaHR0cC1nZXQtZXJyb3ItaGVhZGVycyBjKSkp KQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAod2Fybmlu ZyAoR18gIkdpdEh1YiByYXRlIGxpbWl0IGV4Y2VlZGVkOyBcCj4gK2Rpc2FsbG93aW5nIHJlcXVl c3RzIGZvciB+YSBzZWNvbmRzfiUiKQo+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAoLSByZXNldCAoY2FyIChnZXR0aW1lb2Zk YXkpKSkpCj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIChk aXNwbGF5LWhpbnQgKEdfICJZb3UgY2FuIHdhaXZlIHRoZSByYXRlIGxpbWl0IGJ5Cj4gK3NldHRp bmcgdGhlIEBlbnZ7R1VJWF9HSVRIVUJfVE9LRU59IGVudmlyb25tZW50IHZhcmlhYmxlIHRvIGEg dG9rZW4gb2J0YWluZWQKPiArZnJvbSBAdXJse2h0dHBzOi8vZ2l0aHViLmNvbS9zZXR0aW5ncy90 b2tlbnN9IHdpdGggeW91ciBHaXRIdWIgYWNjb3VudC4KCklJUkMsIHRoZSBHaXRIdWIgZG9jdW1l bnRhdGlvbiBkb2Vzbid0IHN0YXRlIHRoYXQgdGhpcyB3YWl2ZXMgdGhlIHJhdGUKbGltaXQsIHJh dGhlciBpdCBpbmNyZWFzZXMgdGhlIHJhdGUgbGltaXQgYSBsb3QsIGJ1dCB0aGVyZSdzIHN0aWxs IGEKbGltaXQuCgpHcmVldGluZ3MsCk1heGltZS4K --=-PcJijoVb93THoeldWDgw Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMvPxccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7m8xAQCtCloUphtmRR0fYGTzLob60MUI Gb6+4KGnfF85do0zkQD/T7zMh2/Q9PfdvhUTbS3kl0jxH1CnP1MiJtCr7ArofAg= =YkRQ -----END PGP SIGNATURE----- --=-PcJijoVb93THoeldWDgw-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 09:45:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 54241@debbugs.gnu.org, Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16464734926216 (code B ref 54241); Sat, 05 Mar 2022 09:45:02 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:44:52 +0000 Received: from localhost ([127.0.0.1]:47304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQy4-0001cC-Cv for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:44:52 -0500 Received: from albert.telenet-ops.be ([195.130.137.90]:52322) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQQy2-0001c4-SF for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:44:51 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by albert.telenet-ops.be with bizsmtp id 2Zko2700U4UW6Th06ZkoPS; Sat, 05 Mar 2022 10:44:49 +0100 Message-ID: <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 10:44:48 +0100 In-Reply-To: <87tucdqwen.fsf_-_@gnu.org> References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-tlTNn3P+kAqf4bC74gxw" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646473489; bh=GzRpajpEHHcqqsKMOdnfCcGmlyied3+zJ5NUA2Yrfn4=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=XLESYN8VGvIvk8pQNwo0GXnIAl7t+MARUam5SIB5KIDWo3Klu4QNFsQH3vfxWYF/e z4DSmeSf4tA5OHhB6W1iqHPdWXkIgAQ/g+yiEc/7l2zdQ9x8fR/1ch/K8ZJ1x5itK8 h5REGLWT7wXCNrbgtKSDd8QqSV8PHYGS4FUjSoAPU0YwbNUwE08a+i15HW7Tua4pDl thpNBADIRwhcwQloX4ZFjVzJe5mGrIZKDKiI8Wyu2mhr70M72TpBvFe64e+FG4fNcn ySPQtBcAnGcYHc9NZFmiBfx1bHlxQ1r3Wo2mrjjP9iGXqwkbIJA3Pa9NYsh1+hVGI+ KcddTMPPOBQyw== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-tlTNn3P+kAqf4bC74gxw Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op vr 04-03-2022 om 21:45 [+0100]: > My concern is that we=E2=80=99d end up caching one or two little files in > ~/.cache for each candidate package, and (rate limit aside) the overhead > of dealing with the cache might outweigh the benefits.=C2=A0 I=E2=80=99d = rather use > =E2=80=98http-fetch/cached=E2=80=99 for bigger files, like in (guix cve). >=20 > WDYT? If the overhead of caching little files is a concern, then perhaps a SQLite (or GDBM) database could be used instead of the filesystem-based cache? The number of packages in Guix was about 150 000 IIRC, if we assume something around the magnitude of 200 bytes per package, then we end up with about 29 MiB for the entirity of Guix. And there might be some opportunities for compression, reducing this number. Something like this could be left for later though. Greetings, Maxime. --=-tlTNn3P+kAqf4bC74gxw Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMxEBccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rlrAQCEUL9l/X7qI4RTPnAN4QTORipl HbeWWKeUBjBFdritsAEAx5BXSS1xnYcyioOiI845303n1EX1gat09Nm7oVWT2QY= =KduL -----END PGP SIGNATURE----- --=-tlTNn3P+kAqf4bC74gxw-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 09:49:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16464737046729 (code B ref 54241); Sat, 05 Mar 2022 09:49:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:48:24 +0000 Received: from localhost ([127.0.0.1]:47309 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQR1T-0001kT-S4 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:48:24 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:39260) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQR1S-0001kL-4v for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:48:22 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by laurent.telenet-ops.be with bizsmtp id 2ZoM270014UW6Th01ZoMUq; Sat, 05 Mar 2022 10:48:21 +0100 Message-ID: <61869824d6b6f27ee1b8e71456e9e5c21b3dea5e.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 10:48:20 +0100 In-Reply-To: <20220303211444.19928-2-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-HlekIjr+3gOGC4dDw3FP" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646473701; bh=gtfcvPBKAQg1Hsmzo5JTbtpLyL8nTKFpacEv58QpCj4=; h=Subject:From:To:Date:In-Reply-To:References; b=aZ0/+LTcD361Cjft23mE6BR07AyhEL2s3KZhHc8WLSqB82bZeXcS80Vqo1d6ajh+p X1O2sI+Wa4kv93nkObN0SWEsYRHQNVWHIEOCAnE4Iqi2cGsqXWL7nVPJSmfzh0KfPZ XOgR+U9TLiZFnjY0O1s86yPrxDWczpWtdUtpzZFhisQqipq1e0etp3/duqISwCWrk7 0Q3pjWhZKTMjOSNvXbHzVg1+out1s3AwXHoBVRfa1hUoNfXI1zby3bjQWe8zH583Jr yMVLI0+XFcEyqZXCxmn67aR136bxl6tWF0fQ94w3V6MkcGLIl1douOXUbiwTPVNg6c CvO9FmhDzT/LQ== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-HlekIjr+3gOGC4dDw3FP Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: > [...] It would be nice to have some tests for the rate limiting code, in tests/import-github.scm. Greetings, Maxime. --=-HlekIjr+3gOGC4dDw3FP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMx5BccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hoaAQDoOMGqBrglGPX1x5Rir5GU6y7/ Km73KN/Sqb+V7vUXmwEAhmqxf4nwrHaDY/pnLI6fqJ8qxBrFZmLI4EdBRjetfQg= =Cd8J -----END PGP SIGNATURE----- --=-HlekIjr+3gOGC4dDw3FP-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 09:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16464737396784 (code B ref 54241); Sat, 05 Mar 2022 09:49:02 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:48:59 +0000 Received: from localhost ([127.0.0.1]:47313 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQR23-0001lM-2h for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:48:59 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:39260) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQR20-0001kL-Pq for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:48:57 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by laurent.telenet-ops.be with bizsmtp id 2Zow2700E4UW6Th01Zowfh; Sat, 05 Mar 2022 10:48:56 +0100 Message-ID: <5ddbea58affe4bdb9a8c8d138bb25ccac9aca65c.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 10:48:56 +0100 In-Reply-To: <20220303211444.19928-2-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-ur4Gy/UYVgxX5BjEb/Qu" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646473736; bh=GNjlSNiuR+Km0iX/KAFFS7i4hklS/6iQnAvBduvGA0k=; h=Subject:From:To:Date:In-Reply-To:References; b=fm293/JbYgJzwlHMw3iu0wuFnQpNkSwnB2CE88CBLwgmNjtPLjoBIk0L/HHldEOnQ LLeStUWdylgrXC/f0F5CeRZcNZ/SoElAAkQce1i5CYBFcojcsKhvOO/zJWWRNStyJI xsl+qcMOFfvg0j51o76MMajrDtpIzxRxNvJmDXMyfJEFEsLcIJiX517cLUvo70Dm9/ UAPi9cO3D9PT3HaEVSxa6TsUurrv+Qp7Xw7+1AZ+1nDvGlEffvqbE3g02y5cOP8uq6 kwRTIAjFIAQXiwfd0wv6eF12U6XxXDdBFojgi/u16L07u3LPmGIKi2qUs3Xp/Sxyh5 9PvG9pTIvtygA== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-ur4Gy/UYVgxX5BjEb/Qu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: > +(define (update-rate-limit-reset-time! headers) > +=C2=A0 "Update the rate limit reset time based on HEADERS, the HTTP resp= onse > +headers." > +=C2=A0 (match (assq-ref headers 'x-ratelimit-reset) > +=C2=A0=C2=A0=C2=A0 ((=3D string->number (? number? reset)) > +=C2=A0=C2=A0=C2=A0=C2=A0 (set! %rate-limit-reset-time reset) > +=C2=A0=C2=A0=C2=A0=C2=A0 reset) > +=C2=A0=C2=A0=C2=A0 (_ > +=C2=A0=C2=A0=C2=A0=C2=A0 0))) When can this second case happen? Greetings, Maxime. --=-ur4Gy/UYVgxX5BjEb/Qu Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMyCBccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hDMAQD1e/L4koL7hN9AvjEIS+A8cjr/ OEjrzHutBRxbEhIOHQEAgCCT/Hn14dudn3MxVJAHvYHwZCHVm7+xrbK2haZ9Ngw= =vjUY -----END PGP SIGNATURE----- --=-ur4Gy/UYVgxX5BjEb/Qu-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 2/4] import: github: Gracefully handle rate limit exhaustion. Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 09:53:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16464739777207 (code B ref 54241); Sat, 05 Mar 2022 09:53:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 09:52:57 +0000 Received: from localhost ([127.0.0.1]:47326 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQR5s-0001sA-P5 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 04:52:56 -0500 Received: from laurent.telenet-ops.be ([195.130.137.89]:45304) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQR5s-0001s2-4w for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 04:52:56 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by laurent.telenet-ops.be with bizsmtp id 2Zsu2700K4UW6Th01ZsuE1; Sat, 05 Mar 2022 10:52:55 +0100 Message-ID: <2321d1fa880d51a5938b14ab0bd006e966fa8ac1.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 10:52:54 +0100 In-Reply-To: <20220303211444.19928-2-ludo@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-iy0AaxjYoYKm9x1o2Cpa" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646473975; bh=+rli85GU0r6u1vRLYm5Dtbe018kNpfigFgx/FPfwbf8=; h=Subject:From:To:Date:In-Reply-To:References; b=htoWyGHlhEK+1M+akifFzlsN+hfVQWd2R2PcWkHwA85zkUQzsW9c2w/ZXfrjGSpeL qTeMvX7hgSyfklynF/v7LMiPbaz+IWfd9v0CRI582zsLk97z/1ZVRBfYZJsL3ZbcTM r5SLq3EEK58/BiEjBGoEV3xIFrK/IfVwKQmJihwJ/yZ3ztQP+xVbllKUWTRzuA3VWU tHd6c83Z2B3q4JJmPb6OFUVkJWs6494t4Yd80CttMCvYMCGACZ5MWM0tDJAf1f6jyl F6IYZUfykbo1Nn8KyfDHXCXr4vb+RtvvKBQMn7QZZF3m+/EXs49TneuQ4KxduW7Xmz sRWPqOCU9HDFg== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-iy0AaxjYoYKm9x1o2Cpa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: > +(define (request-rate-limit-reached?) > +=C2=A0 "Return true if the rate limit has been reached." > +=C2=A0 (and %rate-limit-reset-time > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (match (< (car (gettimeofday)) %rat= e-limit-reset-time) > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (#t #t) > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (#f > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (set! %rate-limit= -reset-time #f) > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 #f)))) The clocks used by the GitHub server cannot exactly be the clock of the local Guix (at least, not in a realistic setting). WDYT of adding a little margin, accounting for the impossibility of clocks exactly matching and allowing for some clock skew? (< (car (gettimeofday)) (+ [5 minutes] %rate-limit-reset-time)) Greetings, Maxime. --=-iy0AaxjYoYKm9x1o2Cpa Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiMy9hccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7pLaAP9so91eVeGOK0n94JDoUpI3IGvO y8tjJaseJjvG8cnMzgEA4Yn5lGp4unZC1YC6Lh1BNcKAHM7+yjjQRiFiX18chAA= =AxYS -----END PGP SIGNATURE----- --=-iy0AaxjYoYKm9x1o2Cpa-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org, Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465175488043 (code B ref 54241); Sat, 05 Mar 2022 22:00:02 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 21:59:08 +0000 Received: from localhost ([127.0.0.1]:49233 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcQe-00025e-9O for submit@debbugs.gnu.org; Sat, 05 Mar 2022 16:59:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40164) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcQc-00025P-VJ for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 16:59:07 -0500 Received: from [2001:470:142:3::e] (port=36538 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcQW-000247-DE; Sat, 05 Mar 2022 16:59:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=GDMwIVgxPyksL0XlUsK8J0fS8EFz1dD5co4Un69j4uo=; b=Z0BYC784Oq+V0Wfxq/5o DYGb92tnb3Cbkd8zKGiRVkYRwRmBu74ELSVMm63PQ0vsmH917uLxMY5s/dTSO3pwXxi6iOjLbSvKj n4q6UCgrkIbjUG67qdU2LoemwHDiPEHywTLiBWYBfkSqmG5aDTGpjULmV6EzDZeKI2Q/fbGzbZRyq Wx/fx0Ajx17rkf7C5rwxzkClgiYzfiUVfIfTJi35+ddRmWKezG3g9Z9ODESfd5v2QDnHroziUXxoZ NsgN2qsO9QDuErHXOqCnG9ztAVqExhmevV+B1OYSf5egw3sMowFJ1xUWgbJCtCAmxVSozEIBDLwm3 pWFolq/LRKjqrQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50112 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcQV-0000hx-EL; Sat, 05 Mar 2022 16:59:00 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> Date: Sat, 05 Mar 2022 22:58:56 +0100 In-Reply-To: <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 10:44:48 +0100") Message-ID: <87a6e4njrz.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op vr 04-03-2022 om 21:45 [+0100]: >> My concern is that we=E2=80=99d end up caching one or two little files in >> ~/.cache for each candidate package, and (rate limit aside) the overhead >> of dealing with the cache might outweigh the benefits.=C2=A0 I=E2=80=99d= rather use >> =E2=80=98http-fetch/cached=E2=80=99 for bigger files, like in (guix cve). >>=20 >> WDYT? > > If the overhead of caching little files is a concern, then perhaps a > SQLite (or GDBM) database could be used instead of the filesystem-based > cache? The number of packages in Guix was about 150 000 IIRC, if we > assume something around the magnitude of 200 bytes per package, then > we end up with about 29 MiB for the entirity of Guix. And there might > be some opportunities for compression, reducing this number. I think this would be going overboard in terms of complexity :-), and it wouldn=E2=80=99t radically change the run-time overhead (you still potentia= lly have to do an HTTP round trip with =E2=80=98If-Modified-Since=E2=80=99, you= =E2=80=99re just saving a few hundred bytes on the response in the best case.) > Something like this could be left for later though. Yup! Ludo=E2=80=99. From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:01:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465176598314 (code B ref 54241); Sat, 05 Mar 2022 22:01:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:00:59 +0000 Received: from localhost ([127.0.0.1]:49238 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcSR-0002A2-MB for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:00:59 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40552) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcSQ-00029o-CV for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:00:58 -0500 Received: from [2001:470:142:3::e] (port=36596 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcSL-0002W1-38; Sat, 05 Mar 2022 17:00:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=Kz+QPp70/yWg67e5+I0D13r9fdWQ0TMnD1Ajcm0QL+4=; b=Nipk7BsQkDP6dnYOyEnG ngHCk6jYlmNtHQbOATVoIDQ7RDw8mtLNx/Uqx8s9ynTemGltvrqQvx/O9dRn78eMxS3YyRJ7LOncC HPVLKW01Um8oARWaSoQox9ijPMj8Xt4pRQOATCqR+dkqioimIkilmsZNTrBVYiS6uRp9m1eAZ5mMT 4yHqRJux5A4LAbX9ex0nIBnO5HD+S7YRFPmibrUCC2DFKT7YmEuF2MrSPVxhvonQN1mKIB0O2eBNW 7Y5H7kbw4T1qml843GAONX5ijS4BNzmNZZMpxk2P0mfWvoR4x7Lue+32UQQq0WRu+b1/Iq4rHkbDZ 3kYwi8IW14J+7Q==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50114 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcSK-0000fl-7r; Sat, 05 Mar 2022 17:00:52 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <81f212aeddb345789105c8b8b90bce032adb6c1a.camel@telenet.be> Date: Sat, 05 Mar 2022 23:00:50 +0100 In-Reply-To: <81f212aeddb345789105c8b8b90bce032adb6c1a.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 10:35:15 +0100") Message-ID: <875yosnjot.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: >> When running "guix refresh" (with no arguments), the 'github' updater >> gets used until the rate limit has been reached, after which "guix >> refresh" automatically picks up the next valid updater, typically >> 'generic-git'. > > Wouldn't this make "guix refresh" non-deterministic (though this might > be an acceptable trade-off)? Correct. It could be said that =E2=80=98guix refresh=E2=80=99 is =E2=80=9C= non-deterministic=E2=80=9D anyway since its result depends on external services. I think it=E2=80=99s= an acceptable tradeoff. Ludo=E2=80=99. From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:02:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465176968397 (code B ref 54241); Sat, 05 Mar 2022 22:02:02 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:01:36 +0000 Received: from localhost ([127.0.0.1]:49243 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcT1-0002BN-UM for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:01:36 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40706) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcT0-0002BB-SK for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:01:35 -0500 Received: from [2001:470:142:3::e] (port=36620 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcSv-0002gX-IK; Sat, 05 Mar 2022 17:01:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=KWo9gr25OeUSXRbEchpnyWyPym53n7/VOvHs4Led4rE=; b=ljBqNaGpQgX0v3i6mo8t RZbZl7LxbKNq8Ptx5Vs9W4BO0svDktNakP+Yh/+REQtgoNK/IqANRVu2SZLH+kPUg46ovKWFv5jwV nM1P/1X0nE/gstsHtnvM9rum7HCs+mpRcAN0/BUZk6UZvwfPyalOVMkmZV/sf1eY8DPM1dMt4ng9n US8OtuGSOsP2uXFqlYiS2Wx3jQ49VUJ/Q1ZBovJTCSA+gjJ/EI2qHBLVw4gPHtxaD/mXc1Fsjn2fS BHRF7hs4Or+iOEJGbxrpurOd2ZjjqYXGXXW/Ci60byBreTmOn2ifhP1MaVblKhLWS+jCykde2iY7k 3s7ZAMMn7GIooQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50116 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcSv-0002Xs-4l; Sat, 05 Mar 2022 17:01:29 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> Date: Sat, 05 Mar 2022 23:01:27 +0100 In-Reply-To: (Maxime Devos's message of "Sat, 05 Mar 2022 10:37:03 +0100") Message-ID: <871qzgnjns.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) TWF4aW1lIERldm9zIDxtYXhpbWVkZXZvc0B0ZWxlbmV0LmJlPiBza3JpYmlzOg0KDQo+IEx1ZG92 aWMgQ291cnTDqHMgc2NocmVlZiBvcCBkbyAwMy0wMy0yMDIyIG9tIDIyOjE0IFsrMDEwMF06DQo+ PiArwqAgKGFuZCAobm90IChyZXF1ZXN0LXJhdGUtbGltaXQtcmVhY2hlZD8pKQ0KPj4gK8KgwqDC oMKgwqDCoCAoZ3VhcmQgKGMgKFsuLi5dDQo+PiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCA7OyBTZWUNCj4+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDs7 IDxodHRwczovL2RvY3MuZ2l0aHViLmNvbS9lbi9yZXN0L292ZXJ2aWV3L3Jlc291cmNlcy1pbi10 aGUtcmVzdC1hcGkjcmF0ZS1saW1pdGluZz4uDQo+PiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoCAobWF0Y2ggKGFzc3EtcmVmIChodHRwLWdldC1lcnJvci1oZWFkZXJzIGMpDQo+ PiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqAgJ3gtcmF0ZWxpbWl0LXJlbWFpbmluZykNCj4+ICvCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAoI2YNCj4+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgIChyYWlzZSBjKSkNCj4+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCAoKD8gKGNvbXBvc2UgemVybz8gc3RyaW5nLT5udW1iZXIpKQ0KPj4g K8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgKGxldCAoKHJlc2V0ICh1 cGRhdGUtcmF0ZS1saW1pdC1yZXNldC10aW1lIQ0KPj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIChodHRwLWdldC1l cnJvci1oZWFkZXJzIGMpKSkpDQo+PiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqAgKHdhcm5pbmcgKEdfICJHaXRIdWIgcmF0ZSBsaW1pdCBleGNlZWRlZDsgXA0K Pj4gK2Rpc2FsbG93aW5nIHJlcXVlc3RzIGZvciB+YSBzZWNvbmRzfiUiKQ0KPj4gK8KgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgICgt IHJlc2V0IChjYXIgKGdldHRpbWVvZmRheSkpKSkNCj4+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAoZGlzcGxheS1oaW50IChHXyAiWW91IGNhbiB3YWl2ZSB0 aGUgcmF0ZSBsaW1pdCBieQ0KPj4gK3NldHRpbmcgdGhlIEBlbnZ7R1VJWF9HSVRIVUJfVE9LRU59 IGVudmlyb25tZW50IHZhcmlhYmxlIHRvIGEgdG9rZW4gb2J0YWluZWQNCj4+ICtmcm9tIEB1cmx7 aHR0cHM6Ly9naXRodWIuY29tL3NldHRpbmdzL3Rva2Vuc30gd2l0aCB5b3VyIEdpdEh1YiBhY2Nv dW50Lg0KPg0KPiBJSVJDLCB0aGUgR2l0SHViIGRvY3VtZW50YXRpb24gZG9lc24ndCBzdGF0ZSB0 aGF0IHRoaXMgd2FpdmVzIHRoZSByYXRlDQo+IGxpbWl0LCByYXRoZXIgaXQgaW5jcmVhc2VzIHRo ZSByYXRlIGxpbWl0IGEgbG90LCBidXQgdGhlcmUncyBzdGlsbCBhDQo+IGxpbWl0Lg0KDQpHb29k IHBvaW50LCBJ4oCZbGwgYWRqdXN0IHRoZSBtZXNzYWdlIGFjY29yZGluZ2x5Lg0K From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:04:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465178208592 (code B ref 54241); Sat, 05 Mar 2022 22:04:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:03:40 +0000 Received: from localhost ([127.0.0.1]:49249 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcV2-0002EV-E5 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:03:40 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41090) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcV0-0002EH-9c for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:03:38 -0500 Received: from [2001:470:142:3::e] (port=36658 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcUv-0002sh-38; Sat, 05 Mar 2022 17:03:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=IjJSofRVF2rtewtm6wgjJA9hFPgn8VYYKZmM9hY+BFg=; b=XlInZtUpcR65650t57Ko l/FIoKQ2XjslfGAKzjAkKr8KIclnNijcz9F9kfDZKi4dRW2Ds5LOt9yyRED6SOMl176yU4F6kcpYI Af8KoeXSspKUIGKCKWR7trjspi2rVaR4QoNIu/wcN9B8sSJy+H5Gj5+EdqeD3fCylJjKDLw3waeiJ XvAIBOoCKvG//POaH4cx2RcO8I3VslrLpjavf/q3qD/VQRT4qRyG5xRrG/7DscJ6efIOnLaJptnnG 3TzskovhtNWX+T4MEM+jvQ1lLMkukInJD61vmFGyeb5Mm6akfoSumI3KWDzUT2D6XD/u7eI0LyWjM pQreA/kF7suqew==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50118 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcUu-0002j5-JW; Sat, 05 Mar 2022 17:03:32 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <5ddbea58affe4bdb9a8c8d138bb25ccac9aca65c.camel@telenet.be> Date: Sat, 05 Mar 2022 23:03:31 +0100 In-Reply-To: <5ddbea58affe4bdb9a8c8d138bb25ccac9aca65c.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 10:48:56 +0100") Message-ID: <87wnh8m4zw.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: >> +(define (update-rate-limit-reset-time! headers) >> +=C2=A0 "Update the rate limit reset time based on HEADERS, the HTTP res= ponse >> +headers." >> +=C2=A0 (match (assq-ref headers 'x-ratelimit-reset) >> +=C2=A0=C2=A0=C2=A0 ((=3D string->number (? number? reset)) >> +=C2=A0=C2=A0=C2=A0=C2=A0 (set! %rate-limit-reset-time reset) >> +=C2=A0=C2=A0=C2=A0=C2=A0 reset) >> +=C2=A0=C2=A0=C2=A0 (_ >> +=C2=A0=C2=A0=C2=A0=C2=A0 0))) > > When can this second case happen? I don=E2=80=99t know if it=E2=80=99s supposed to happen. It=E2=80=99s defe= nsive programming: better keep going than crash if the server starts behaving slightly differently. From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:05:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 54241@debbugs.gnu.org, Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465178918710 (code B ref 54241); Sat, 05 Mar 2022 22:05:02 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:04:51 +0000 Received: from localhost ([127.0.0.1]:49254 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcWA-0002GQ-P3 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:04:50 -0500 Received: from andre.telenet-ops.be ([195.130.132.53]:42708) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcW8-0002GH-Fy for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:04:49 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by andre.telenet-ops.be with bizsmtp id 2m4n270014UW6Th01m4n58; Sat, 05 Mar 2022 23:04:47 +0100 Message-ID: <7d4255f8670a135420ff526262459035adddb0d5.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 23:04:43 +0100 In-Reply-To: <87a6e4njrz.fsf_-_@gnu.org> References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> <87a6e4njrz.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-SPnaPPMm3kHU0mKlTq7q" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646517887; bh=OMIC92/BSed0K8FZYPhuWtG025mIC9WyT1to1t89+FI=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=i5zEcixMoZxSNTCF7gr7YQ8tKYiHLCDdm7+CE4H750hjn3YxBv4smZwP5XXMBmm4Q 3rrek0VnIwFo9a0FA+u3f/929uPqlJlutcupeWnD2cqfCqZax3/Hg+JJhABMa1tQds asbxS9MgHhd1BzxsW/RZwyB/TFJlfOLqUSGeZdiLFljXOH4f4U8hh2EBOoHw4IV1e5 6og2rLn7z9Jq4R7/E8Q5uiqt9kuFSJixcW+ZM7dSlR66g1LdGbHKYPUcZmN5KUs6eW bP7Hu+60xWHx6yry70LK+kzRBJZi+hXyxqK56uMDxs6Qby1SgK7Q5hVxETTFTxR3Zx BLKR9kN6IEM8A== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-SPnaPPMm3kHU0mKlTq7q Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op za 05-03-2022 om 22:58 [+0100]: > [...] and it wouldn=E2=80=99t radically change the run-time overhead (you= still > potentially have to do an HTTP round trip with =E2=80=98If-Modified-Since= =E2=80=99, > you=E2=80=99re just saving a few hundred bytes on the response in the bes= t case.) IIUC, when the TTL hasn't been exceeded, then the file from the file system is served without contacting the remote server at all. So in the best case, you only =E2=80=98round-trip=E2=80=99 to the disk instead of= the HTTP server. So I think there's some potential benefits to be had here. That assumes a sufficiently large TTL though. Greetings, Maxime. --=-SPnaPPMm3kHU0mKlTq7q Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiPeexccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7vUcAP94Itr7wlL8pLLDKKx7dR1VYbKd WKECqLQqpwntYDY/CwD8CdKWcCfsYnIcYQNlJsgn0WwmxVIFyUUrYHoQU5W5XA0= =cbRT -----END PGP SIGNATURE----- --=-SPnaPPMm3kHU0mKlTq7q-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:07:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465179908884 (code B ref 54241); Sat, 05 Mar 2022 22:07:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:06:30 +0000 Received: from localhost ([127.0.0.1]:49259 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcXm-0002JE-2c for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:06:30 -0500 Received: from eggs.gnu.org ([209.51.188.92]:41930) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcXk-0002J0-NW for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:06:28 -0500 Received: from [2001:470:142:3::e] (port=36692 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcXf-0003Wc-IT; Sat, 05 Mar 2022 17:06:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=6PSUAEg6m/9m1ccQXjedQOeKynvTP/CzzRoo+cg9h3s=; b=r50j6pU/1jzhlhtEE/ST rx6Y4hqs4jXXtTzQF0La07IBHMdk3n2uTa9LMW4oGvjJC6kfvr6mF34NcO1wcoC6JckVujWBZm2HP q6UtNh5cszogHvAqgPHLBjUXPjJTEDji81bKTeiS0W7LKFqkwapSds6jUCEMuLMmmHYB8kqrk4hDX Bqxw/sxpjqzXTHJmvi9RwRAaHYchGpZ0IN5l7durwpeIQwRVFnZpf4aYz7YqoJif0qtNxR6gLGB8u q9FLmvCKB3LgEFnRSviX8BEVfSqE6ihNyd8mPlppGkwu+P05HjTszvhjlFWMhdPhxImCcsT7RDqsG BGpwfrbY087vAg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50120 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcXf-00034V-5L; Sat, 05 Mar 2022 17:06:23 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <2321d1fa880d51a5938b14ab0bd006e966fa8ac1.camel@telenet.be> Date: Sat, 05 Mar 2022 23:06:21 +0100 In-Reply-To: <2321d1fa880d51a5938b14ab0bd006e966fa8ac1.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 10:52:54 +0100") Message-ID: <87r17gm4v6.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: >> +(define (request-rate-limit-reached?) >> +=C2=A0 "Return true if the rate limit has been reached." >> +=C2=A0 (and %rate-limit-reset-time >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (match (< (car (gettimeofday)) %ra= te-limit-reset-time) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (#t #t) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (#f >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (set! %rate-limi= t-reset-time #f) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 #f)))) > > The clocks used by the GitHub server cannot exactly be the clock of the > local Guix (at least, not in a realistic setting). WDYT of adding a > little margin, accounting for the impossibility of clocks exactly > matching and allowing for some clock skew? > > (< (car (gettimeofday)) (+ [5 minutes] %rate-limit-reset-time)) I don=E2=80=99t think it=E2=80=99s necessary. The worst that can happen is= that we retry too early, get another 403 response, and retry later. (In practice, on my NTP-synchronized laptop, things just work.) From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465181669145 (code B ref 54241); Sat, 05 Mar 2022 22:10:02 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:09:26 +0000 Received: from localhost ([127.0.0.1]:49264 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcac-0002NQ-IU for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:09:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42314) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcab-0002NE-70 for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:09:25 -0500 Received: from [2001:470:142:3::e] (port=36734 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcaW-0003o4-1m; Sat, 05 Mar 2022 17:09:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=pKVN1NjI4QfXbOfwm7gc2wY6wrSCsqS36FEIjLOQqb0=; b=hu1DjNsVXN6dO3PyPUOH JU2hTOwpBxoygX1uslySPfC/GnIoLPCil+vNGAkLa697Y9162m1XfGi9cnU6uemPJwWMm1y/2Fau1 kA5x4lilrmWB9ha1l5fMoHqGFn8NjsapMDkn+Lqufgxu61jPlU3L4apbzRgAxG2OGAXinsT/L7ZcK DWwJ+Lffjo3EsB1btR+NNkXD0knK1xhR62nM9zZokCUtQ8WxHlPpHRKJYmJCn7bAmBVaUEo9X4Mr0 FotUf8q03DW31A7fNZcoDt2QGx2b0gZ62WMyhxPAGX+qEncsfzOZFm4vjaZKmn5cnLPnc8mxN/xVX qZFDM+kS7bsR1g==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50122 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQcaV-0003DX-HR; Sat, 05 Mar 2022 17:09:19 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <61869824d6b6f27ee1b8e71456e9e5c21b3dea5e.camel@telenet.be> Date: Sat, 05 Mar 2022 23:09:18 +0100 In-Reply-To: <61869824d6b6f27ee1b8e71456e9e5c21b3dea5e.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 10:48:20 +0100") Message-ID: <87mti4m4q9.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Maxime Devos skribis: > It would be nice to have some tests for the rate limiting code, > in tests/import-github.scm. It would but=E2=80=A6 I think I=E2=80=99ll punt on that one. :-) From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:12:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 54241@debbugs.gnu.org, Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465182709321 (code B ref 54241); Sat, 05 Mar 2022 22:12:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:11:10 +0000 Received: from localhost ([127.0.0.1]:49269 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQccH-0002QH-T0 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:11:10 -0500 Received: from baptiste.telenet-ops.be ([195.130.132.51]:52746) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQccF-0002Q8-Sn for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:11:08 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by baptiste.telenet-ops.be with bizsmtp id 2mB6270034UW6Th01mB6ej; Sat, 05 Mar 2022 23:11:06 +0100 Message-ID: <48d960ad6ec09604d9271cfae7d57796b564e1c0.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 23:11:05 +0100 In-Reply-To: <87a6e4njrz.fsf_-_@gnu.org> References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> <87a6e4njrz.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-L3SdffvcGAD73eY1/qdv" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646518266; bh=fjN7SJzOKk/NU9JgWZLc0K5nlOAV0ZX1YI2yAiWJi5g=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=QS7tReiVXbcUADnt9AiEZB8ZaXXCd4LNtRD7HauuV0QkCcFZWytiXpVGHfTFgvoAR cEoBqLO5XAitbFeLMoxyYZemFx9xuk9RIqyjWIvlgqFZ9hrqyHmEH2h/voWVasbI9j gI3VTEQCCJDfnZ2Y0Tnf/3krbEIF16MiaHAvdZs62BYJSJGw42xy1RK97cR9UZ79n7 89HdG3LPrYTMGlmQFJVxaLFKwNlcLFqwyV/qUROTu+VraYcbJb/mW7CMqUq4m+bMCf DVwpIOc7hNw7VmmFsy/1zDTZvJzNt3seHVvJC5cHMwUW1Stzb1LxjF0kPlIoBVxqrr nmu2k1uKbDM2w== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-L3SdffvcGAD73eY1/qdv Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op za 05-03-2022 om 22:58 [+0100]: > I think this would be going overboard in terms of complexity :-) There's some complexity here, but assuming a sufficient amount of tests, I believe it would be worth it if it allows side-stepping the rate limit to some degree. And the extra complexity would mostly disappear if the overhead of tiny files was accepted (*). There are also some other benefits, e.g. a kind of =E2=80=98download resumption=E2=80=99 but for linters, reducing network traffic after retryin= g "guix lint" on a lossy network (or because the terminal tab was closed to early, etc.). All stuff that can be left for later though! Greetings, Maxime. (*) Assuming 150 000 packages and 1 KiB per package (this would be file-system dependent!), I end up with 150 MiB. That's a bit on the large size though ... --=-L3SdffvcGAD73eY1/qdv Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiPf+RccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7huQAP9s5wWhgz3jLsn+2DSC3Atx+4Xq eWeeg2PFGrMeFBIYlwEA+2iz94aC0r3JK607TD53EzXK1LYv52tp8nkdrKbnHws= =3jco -----END PGP SIGNATURE----- --=-L3SdffvcGAD73eY1/qdv-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.164651856714023 (code B ref 54241); Sat, 05 Mar 2022 22:17:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:16:07 +0000 Received: from localhost ([127.0.0.1]:49289 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQch5-0003dt-B4 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:16:07 -0500 Received: from baptiste.telenet-ops.be ([195.130.132.51]:56030) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQch3-0003c7-Ck for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:16:05 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by baptiste.telenet-ops.be with bizsmtp id 2mG4270084UW6Th01mG40u; Sat, 05 Mar 2022 23:16:04 +0100 Message-ID: From: Maxime Devos Date: Sat, 05 Mar 2022 23:16:04 +0100 In-Reply-To: <87wnh8m4zw.fsf_-_@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <5ddbea58affe4bdb9a8c8d138bb25ccac9aca65c.camel@telenet.be> <87wnh8m4zw.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-8crGG47xShQud94oHg+n" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646518564; bh=keOs7amHy1dw8EAY74ZreQr/Q6jWvJFSKxJZ4cgo7Vk=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=dcF+lc6R3cDgdtdMqfr62UWZRfIA3ODi22OSU+B6LbH2KDsZklhublcZlraCe6jeI xTrldPxPryzEMllPwjl/qBe8U9QDthvAWzVsJujfamxWnvbT3QavXlY1e4xb24bloJ hqAVycg3Xdp6hJzxvWw9JR70E/Bu4dOJIszQ5LkopSqlL2wLpHopmF2VW968kDh9ns BoNf+Ljb2emywTgT4D5DQCxrNY1YMdLjDBCvagAPLCzhZ8vvuwlahTyJLOHn0mHI9j mS5S8Z5wMadE+/oLRdnaQFRWwbtVj0Ji+ID+KkxM0P6BcAyX0wKjK7P8YvG8wyClIT xmK7Za9yKliRQ== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-8crGG47xShQud94oHg+n Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op za 05-03-2022 om 23:03 [+0100]: > I don=E2=80=99t know if it=E2=80=99s supposed to happen.=C2=A0 It=E2=80= =99s defensive programming: > better keep going than crash if the server starts behaving slightly > differently. That's called total programming I think? From a OOP I'm following: * total: handle all cases without complaints (no throwing exceptions or such), assign every case a well-defined (and documented!) behaviour * nominal: document the preconditions, but don't bother checking them * defensive: check inputs, if they are wrong, throw an exception (it was probably formulated a bit differently but that was the gist of it) At least according to this classification, this 'update-rate-limit-reset-time!' would be total (except for the lack of documentation), not defensive. Greetings, Maxime. --=-8crGG47xShQud94oHg+n Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiPhJBccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7s2AAP90etfPQ3L09auXAbp+Kv+8/1Q/ euTM5+gp29o6M+AazQEAg66z8EnIYmmKIFYD2kEjEdmRm3R6auhhmIbgIpmiQwk= =/aVa -----END PGP SIGNATURE----- --=-8crGG47xShQud94oHg+n-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 05 Mar 2022 22:22:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.164651886718776 (code B ref 54241); Sat, 05 Mar 2022 22:22:01 +0000 Received: (at 54241) by debbugs.gnu.org; 5 Mar 2022 22:21:07 +0000 Received: from localhost ([127.0.0.1]:49305 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQclv-0004sm-J5 for submit@debbugs.gnu.org; Sat, 05 Mar 2022 17:21:07 -0500 Received: from albert.telenet-ops.be ([195.130.137.90]:48878) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQcls-0004sd-T2 for 54241@debbugs.gnu.org; Sat, 05 Mar 2022 17:21:06 -0500 Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a]) by albert.telenet-ops.be with bizsmtp id 2mM32700H4UW6Th06mM3aT; Sat, 05 Mar 2022 23:21:03 +0100 Message-ID: <73b2e8d49a3c178d06e2d7c2984f499a66e249d4.camel@telenet.be> From: Maxime Devos Date: Sat, 05 Mar 2022 23:21:03 +0100 In-Reply-To: <87wnh8m4zw.fsf_-_@gnu.org> References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <5ddbea58affe4bdb9a8c8d138bb25ccac9aca65c.camel@telenet.be> <87wnh8m4zw.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-1G6dmZYQg1NLG/y78wVd" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22; t=1646518863; bh=oUs+kG9q0e9Zif+SqGeb/M5japuTcwTdTJf+OQZYCHE=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=WpYV8L4TSAOdql6zCeD1K+4DKJwagwtgdf/FW9OxwOzpaaknXmBHeyhLfWhg6GN7A G6ouQWHbEmHFYyhZsy45CCCTFa8jIoEMTyGwB68vP9Lz4XUs/UCMRVwVScuEvGIaeB pRqAbgerwyRkQbsXT2IW7HG2L4Z8bBWeoeCq43hVytlcjsXI51euFrLrhP9xZpaQtJ LZomoY6jU1EjZ3WB+NQtfqhIHOHpsxtYukKegQQ8R3jRt6P298yrl15UeMcC/UnAkw 79T1Q7ynXeuiIdWJAa6cyrPAOwzGn2tDpOM2Kxp393WvBcFrvMTT9+b6r8pi+g4DPe mLvd0dPlcU1ig== X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-1G6dmZYQg1NLG/y78wVd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op za 05-03-2022 om 23:03 [+0100]: > Maxime Devos skribis: >=20 > > Ludovic Court=C3=A8s schreef op do 03-03-2022 om 22:14 [+0100]: > > > +(define (update-rate-limit-reset-time! headers) > > > +=C2=A0 "Update the rate limit reset time based on HEADERS, the HTTP = response > > > +headers." > > > +=C2=A0 (match (assq-ref headers 'x-ratelimit-reset) > > > +=C2=A0=C2=A0=C2=A0 ((=3D string->number (? number? reset)) > > > +=C2=A0=C2=A0=C2=A0=C2=A0 (set! %rate-limit-reset-time reset) > > > +=C2=A0=C2=A0=C2=A0=C2=A0 reset) > > > +=C2=A0=C2=A0=C2=A0 (_ > > > +=C2=A0=C2=A0=C2=A0=C2=A0 0))) > >=20 > > When can this second case happen? >=20 > I don=E2=80=99t know if it=E2=80=99s supposed to happen.=C2=A0 It=E2=80= =99s defensive programming: > better keep going than crash if the server starts behaving slightly > differently. If it's not supposed to happen, can it at least be reported with a warning, such that=C2=A0we then know that 'update-rate-limit-reset-time!' needs to be extended or GitHub needs to be contacted? FWIW, I think crashing in case of bogus HTTP answers is fine, as long as it crashes with a _nice_ error message ("guix: error: HTTP server foo.com returned an unrecoginised X-Ratelimit-Reset $SOME_STRING" or something like that) instead of some vague backtrace. Greetings, Maxime. --=-1G6dmZYQg1NLG/y78wVd Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYiPiTxccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7iaOAQDLWPqjDOJIM5ceCo62t3ELxIHu burvTnsk3Y+pMnYLMwD+InZCjUVt40CTioWzHL5moDsQZbljmV7BXn1/SOj5nwQ= =3JJp -----END PGP SIGNATURE----- --=-1G6dmZYQg1NLG/y78wVd-- From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 06 Mar 2022 17:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465871024473 (code B ref 54241); Sun, 06 Mar 2022 17:19:01 +0000 Received: (at 54241) by debbugs.gnu.org; 6 Mar 2022 17:18:22 +0000 Received: from localhost ([127.0.0.1]:51650 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQuWU-0001A5-DE for submit@debbugs.gnu.org; Sun, 06 Mar 2022 12:18:22 -0500 Received: from eggs.gnu.org ([209.51.188.92]:49524) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQuWR-00019s-R6 for 54241@debbugs.gnu.org; Sun, 06 Mar 2022 12:18:20 -0500 Received: from [2001:470:142:3::e] (port=52486 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQuWM-00071Q-FB; Sun, 06 Mar 2022 12:18:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=64Z1KDXZOCVqTXpAjqggw3U0zQd5+5nrj/CnHGhhzmY=; b=CkxZ2HWnXP8JNpWTsVrj 1x7OVUOsFYz/hxsVhQiemIyyyUZ1MrVF+NxnuBVJnuyVfOpfppDzPgBFLV89ghZ+aJdC8mFksbgiN LY68/uv8AJHLY0kvIQZMcTB0h6gQCsuJSFbi01g1VI9S87GxFh+N+oH0+sJto5zMZPCVeKa41X9sf N35T02F/ucDhrTBdpMHx5f0fd2YASKZVeadyMrFPJfro4nEgSqLGYnlwBq/I5p9lfscgBNTwhrSyp adPf5By9R6xHpmI1FYfQBcKCvA4EDRwxHttkxzJvTROnw9qglwTNMr1R6ZP7vyI6gxnuq46eRO0Jc lD4xYCOsCXg6yA==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50136 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQuWL-000540-Gy; Sun, 06 Mar 2022 12:18:13 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-2-ludo@gnu.org> <5ddbea58affe4bdb9a8c8d138bb25ccac9aca65c.camel@telenet.be> <87wnh8m4zw.fsf_-_@gnu.org> Date: Sun, 06 Mar 2022 18:18:10 +0100 In-Reply-To: (Maxime Devos's message of "Sat, 05 Mar 2022 23:16:04 +0100") Message-ID: <874k4bknjh.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Maxime Devos skribis: > That's called total programming I think? From a OOP I'm following: > > * total: handle all cases without complaints (no throwing exceptions > or such), assign every case a well-defined (and documented!) > behaviour > * nominal: document the preconditions, but don't bother checking them > * defensive: check inputs, if they are wrong, throw an exception Interesting; I stand corrected! > If it's not supposed to happen, can it at least be reported with a > warning, such that=C2=A0we then know that 'update-rate-limit-reset-time!' > needs to be extended or GitHub needs to be contacted? Yes, sounds reasonable. Thanks, Ludo=E2=80=99. From unknown Sun Jun 22 07:55:05 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#54241] [PATCH 0/4] 'github' importer gracefully handles rate limiting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 06 Mar 2022 17:22:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 54241 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: 54241@debbugs.gnu.org, Nicolas Goaziou Received: via spool by 54241-submit@debbugs.gnu.org id=B54241.16465873064797 (code B ref 54241); Sun, 06 Mar 2022 17:22:01 +0000 Received: (at 54241) by debbugs.gnu.org; 6 Mar 2022 17:21:46 +0000 Received: from localhost ([127.0.0.1]:51656 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQuZj-0001FH-Ts for submit@debbugs.gnu.org; Sun, 06 Mar 2022 12:21:46 -0500 Received: from eggs.gnu.org ([209.51.188.92]:49918) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQuZh-0001F4-H8 for 54241@debbugs.gnu.org; Sun, 06 Mar 2022 12:21:42 -0500 Received: from [2001:470:142:3::e] (port=52558 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQuZb-0007hS-Nj; Sun, 06 Mar 2022 12:21:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=A6bilawvI9TGpgqRmpSAc0ShJK3mj+iL/vRjMRN3baw=; b=QqAhuUp/2gmMBlMya1Oo IT3EWH1o80/vReYux+3nhOorxVyBAdXWWEORStCkBcEKivyyDl59jF6Vs2AO6kzPm+gCrphURdyk+ DpXXWEZWidVGJrAq3jxdsg+Qap9ZpQFtFpjidrmfYVXFrd02Nq8wgwgoa2JWIY3jFjifNaBSys4mC 6W3h2xAsR4hie+TDEdAuLu9chGV5DNCgDJSDY378F4UF+mQ1Z7ATO7Y63zS9GjxsP540rYj1d/v7T 9z9q07//lESbXzWvS/HbwrPmZlvWd3Wpnd0BJnPLZbruYza2RF/zFtBakS/XMrJX6aWMjLhqoQHpp LhBwtir7B9B9fw==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50138 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQuZb-0005Rg-9C; Sun, 06 Mar 2022 12:21:35 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20220303211326.19884-1-ludo@gnu.org> <95b755ed648ffc15d7dcedb09f538a026fcfeb10.camel@telenet.be> <87tucdqwen.fsf_-_@gnu.org> <9ec3d594aaf4e2ed835a15999c07398bb7c96029.camel@telenet.be> <87a6e4njrz.fsf_-_@gnu.org> <48d960ad6ec09604d9271cfae7d57796b564e1c0.camel@telenet.be> Date: Sun, 06 Mar 2022 18:21:33 +0100 In-Reply-To: <48d960ad6ec09604d9271cfae7d57796b564e1c0.camel@telenet.be> (Maxime Devos's message of "Sat, 05 Mar 2022 23:11:05 +0100") Message-ID: <87zgm3j8te.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Maxime Devos skribis: > Ludovic Court=C3=A8s schreef op za 05-03-2022 om 22:58 [+0100]: >> I think this would be going overboard in terms of complexity :-) > > There's some complexity here, but assuming a sufficient amount of > tests, I believe it would be worth it if it allows side-stepping the > rate limit to some degree. What should also be taken into account is the usefulness of the =E2=80=98gi= thub=E2=80=99 updater=E2=80=94investment should be proportionate. I suspect it=E2=80=99s= much less useful now that we have the =E2=80=98generic-git=E2=80=99 updater. Maybe, = maybe it gives slightly more accurate data in some cases, maybe it can be slightly faster, but that=E2=80=99s not entirely clear to me. From unknown Sun Jun 22 07:55:05 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Subject: bug#54241: closed (Re: bug#54241: [PATCH 0/4] 'github' importer gracefully handles rate limiting) Message-ID: References: <875yoqiw46.fsf_-_@gnu.org> <20220303211326.19884-1-ludo@gnu.org> X-Gnu-PR-Message: they-closed 54241 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 54241@debbugs.gnu.org Date: Sun, 06 Mar 2022 21:57:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1646603822-6706-1" This is a multi-part message in MIME format... ------------=_1646603822-6706-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #54241: [PATCH 0/4] 'github' importer gracefully handles rate limiting which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 54241@debbugs.gnu.org. --=20 54241: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D54241 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1646603822-6706-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 54241-done) by debbugs.gnu.org; 6 Mar 2022 21:56:03 +0000 Received: from localhost ([127.0.0.1]:51902 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQyrD-0001iX-20 for submit@debbugs.gnu.org; Sun, 06 Mar 2022 16:56:03 -0500 Received: from eggs.gnu.org ([209.51.188.92]:59898) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nQyrC-0001i2-8S for 54241-done@debbugs.gnu.org; Sun, 06 Mar 2022 16:56:02 -0500 Received: from [2001:470:142:3::e] (port=58058 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQyr6-0004GN-GI; Sun, 06 Mar 2022 16:55:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=WyJcQQa7Or3esmyjZzqNXrsUzEne8w3GHeTQdqzGV4k=; b=HUmr4LvOknHWHTEGov2W KITLaNO1APlxaQd+2jWBpqEBcsV8krbJI4EW8hNlplzaSZeCu76ukIjKIRTHEtJCp+zK0LGYVVjaw KtyjdPaG917mYuDM8nCV8zmvC+21tqmUN9BalSllJ8w8L4KwppZERHwz4nrIFTe3aaHgRIHv1oC7c rgfs/2q1zt3x09O9AjCUh4mxN2y0Smzg8gpPjX6dnlONLhc+ceyxhbVYv0nxPTfvXfCux58A3EoEk 9SHNISkb7Q8+vEoeBKHtkso4GARXCTkFN5cdAp4U5eu7jP6GDVSw8gju7tUzDgjYQrhXKsy6GBTww C66J2Bo4YvbzXQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50164 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nQyr6-0007vJ-2j; Sun, 06 Mar 2022 16:55:56 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxime Devos Subject: Re: bug#54241: [PATCH 0/4] 'github' importer gracefully handles rate limiting References: <20220303211444.19928-1-ludo@gnu.org> <20220303211444.19928-3-ludo@gnu.org> <69a48935f3f03ad72ac49eb7b529cf10eb7db96e.camel@telenet.be> Date: Sun, 06 Mar 2022 22:55:53 +0100 In-Reply-To: <69a48935f3f03ad72ac49eb7b529cf10eb7db96e.camel@telenet.be> (Maxime Devos's message of "Fri, 04 Mar 2022 13:39:05 +0100") Message-ID: <875yoqiw46.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 54241-done Cc: 54241-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, I committed an updated version of these patches, taking some of your suggestions into account: a8d3033da6 import: github: Reuse HTTP connection for the /tags URL fallba= ck. 8786c2e8d7 http-client: Correctly handle redirects when #:keep-alive? #t. 55e8e283ae import: github: Gracefully handle rate limit exhaustion. ecad9b2213 http-client: Add response headers to '&http-get-error'. 049aefddb2 tests: Add (guix http-client) tests. The first patch adds tests for =E2=80=98http-fetch=E2=80=99, as you suggest= ed, which allowed me to find a thinko in the keep-alive patch. The test doesn=E2=80= =99t test keep-alive support though, because the server in (guix tests http) doesn=E2=80=99t support it currently (I tried to retrofit it based on (web server http) but that=E2=80=99s not really possible because we=E2=80=99d ne= ed to either access internals of the record type, specifically it=E2=80=99s poll = set, or re-implement it entirely.) Thanks a lot for your help! Ludo=E2=80=99. ------------=_1646603822-6706-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 3 Mar 2022 21:13:38 +0000 Received: from localhost ([127.0.0.1]:43923 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPslW-0002BD-EZ for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:13:38 -0500 Received: from lists.gnu.org ([209.51.188.17]:56718) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nPslT-0002B4-Vl for submit@debbugs.gnu.org; Thu, 03 Mar 2022 16:13:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34888) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPslT-0000DC-PJ for guix-patches@gnu.org; Thu, 03 Mar 2022 16:13:35 -0500 Received: from [2001:470:142:3::e] (port=58802 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPslT-0006ny-Co; Thu, 03 Mar 2022 16:13:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=ZhT0X9ZeOlFlFSMn0oLBwuMcsNNYusHsyznJj450XNw=; b=r2MQLtOj9yNVA1 bD3y613EqoKHzjDBJQG15P+sFLDWDPReUuAJiH8eMYxhA9ektw+/tpR8AIA9Y/d9CkYSmtj91Yl92 eQlwNd6Upn2xPCyMSPyt1tG+8/qdrnsQExqAU3QZ3gjC731gXGkgRQgRs4tHt7HKGMfxe9h4wcuas fdQkdRJ+712RZJVuycN8+r8RsMFx1+m/mCqu/wGYSHNmzMpC26uRr0Z0lD/1be6lExckYZ9iU8hDV n0L/RymWGqKJkwHg9QssKMLvLW9LN6ozXwpMBBoqVXlT/chSUL7X4JVGlh3l28rLBRBwpoZ3N99bt IEx3vnLv8ih20zB/yNtg==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201]:50078 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1nPslS-0006DC-Ut; Thu, 03 Mar 2022 16:13:35 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH 0/4] 'github' importer gracefully handles rate limiting Date: Thu, 3 Mar 2022 22:13:26 +0100 Message-Id: <20220303211326.19884-1-ludo@gnu.org> X-Mailer: git-send-email 2.34.0 X-Debbugs-Cc: Nicolas Goaziou , MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Guix! These patches address a famous complaint about “the GitHub problem” that affects ‘guix refresh’¹, shown here in its naked awfulness: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix refresh gnu/packages/zig.scm:32:13: zig would be upgraded from 0.9.0 to 0.9.1 [...] In guix/scripts/refresh.scm: 578:14 5 (_ _) In srfi/srfi-1.scm: 634:9 4 (for-each # _) In guix/scripts/refresh.scm: 378:2 3 (check-for-package-update # (#<) …) In guix/import/github.scm: 232:12 2 (latest-release _) In ice-9/boot-9.scm: 1685:16 1 (raise-exception _ #:continuable? _) 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: Error downloading release information through the GitHub API. This may be fixed by using an access token and setting the environment variable GUIX_GITHUB_TOKEN, for instance one procured from https://github.com/settings/tokens --8<---------------cut here---------------end--------------->8--- With this change, ‘guix refresh’ warns you when the GitHub rate limit is reached, but it keeps going, falling back to the ‘generic-git’ updater if it’s among the applicable updaters: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix refresh -t github,generic-git [...] guix refresh: warning: GitHub rate limit exceeded; disallowing requests for 1477 seconds hint: You can waive the rate limit by setting the `GUIX_GITHUB_TOKEN' environment variable to a token obtained from `https://github.com/settings/tokens' with your GitHub account. Alternatively, you can wait until your rate limit is reset, or use the `generic-git' updater instead. gnu/packages/zile.scm:113:14: warning: no tags were found for zile-on-guile gnu/packages/zig.scm:32:13: zig would be upgraded from 0.9.0 to 0.9.1 gnu/packages/xorg.scm:2830:7: warning: no valid tags found for xf86-video-freedreno gnu/packages/xml.scm:2132:13: java-kxml2 would be upgraded from 2.4.2 to 2.5.0 --8<---------------cut here---------------end--------------->8--- The GitHub updater becomes functional again once the rate limit has been reset. The code to deal with rate limiting is similar to that in (guix swh). Thoughts? Thanks, Ludo’. ¹ https://issues.guix.gnu.org/53818#50 Ludovic Courtès (4): http-client: Add response headers to '&http-get-error'. import: github: Gracefully handle rate limit exhaustion. http-client: Correctly handle redirects when #:keep-alive? #t. import: github: Reuse HTTP connection for the /tags URL fallback. .dir-locals.el | 1 + guix/http-client.scm | 39 +++++++++----- guix/import/github.scm | 119 +++++++++++++++++++++++++++++------------ 3 files changed, 112 insertions(+), 47 deletions(-) base-commit: be84fb701bf7a36a0eb50147ccbb988aa3f41209 -- 2.34.0 ------------=_1646603822-6706-1--