From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix@cbaines.net, dev@jpoiret.xyz, ludo@gnu.org, othacehe@gnu.org, rekado@elephly.net, zimon.toutoune@gmail.com, me@tobias.gr, guix-patches@gnu.org Resent-Date: Fri, 20 Oct 2023 16:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 66650@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 65720@debbugs.gnu.org, Josselin Poiret , Simon Tournier , Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Christopher Baines , Josselin Poiret , Ludovic =?UTF-8?Q?Court=C3=A8s?= , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Received: via spool by submit@debbugs.gnu.org id=B.169781857317976 (code B ref -1); Fri, 20 Oct 2023 16:17:01 +0000 Received: (at submit) by debbugs.gnu.org; 20 Oct 2023 16:16:13 +0000 Received: from localhost ([127.0.0.1]:41305 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qtsAW-0004fs-VW for submit@debbugs.gnu.org; Fri, 20 Oct 2023 12:16:13 -0400 Received: from lists.gnu.org ([2001:470:142::17]:49828) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qtsAV-0004f0-F5 for submit@debbugs.gnu.org; Fri, 20 Oct 2023 12:16:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qts9t-0008NU-Dy for guix-patches@gnu.org; Fri, 20 Oct 2023 12:15:33 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qts9r-0004Rf-PD; Fri, 20 Oct 2023 12:15:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=guykA3Jje4B/fdh8d1YKTVE8F+9iXXIl3FUKarfygvY=; b=jF0iDL/GbSiYleynVOTG /8r/H3EkXts9gzZHBgd9R8nNRQDQE9k6G5S40965qhyaaKSTNmvZmEP05W378rPNL287CkVM+Ibc7 7E32AmsdJM3Am6miqMoPX9YkU5o+Q5z+orHS8Uflfk+Q6GcnHJFgV54Bg6XcAw2I1w6n8BhwLD8XH Xc6+syD3NXx+2uTgwx7sgIiSqx8Q8TyYiO+e+VCDOrgg9PMq1m8eYZ6M32W5h/fNQYw3CoJwQsjTq N7T/JL+bxKLs4dAkgwLr8izbta894kjMEC5GTNjZ2vBkHDJJ991iz+BbSwyZrUfYucrv//2aKtBun iam1wsZbcvK/hA==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 20 Oct 2023 18:15:12 +0200 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <87jzswsrlt.fsf@gnu.org> References: <87jzswsrlt.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Fixes . This fixes a bug whereby libgit2-managed checkouts would keep growing as we fetch. * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New procedures. (update-cached-checkout): Use it. --- guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) Hi! This is a radical fix/workaround for the unbounded Git checkout growth problem, shelling out to ‘git gc’ when it’s likely needed (“too many” pack files around). I thought we might be able to implement a ‘git gc’ approximation using the libgit2 “packbuilder” interface, but I haven’t got around to doing it: . Once again, shelling out is not my favorite option, but it’s a bug we should fix sooner rather than later, hence this compromise. Thoughts? Ludo’. diff --git a/guix/git.scm b/guix/git.scm index b7182305cf..d704b62333 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2020 Mathieu Othacehe -;;; Copyright © 2018-2022 Ludovic Courtès +;;; Copyright © 2018-2023 Ludovic Courtès ;;; Copyright © 2021 Kyle Meyer ;;; Copyright © 2021 Marius Bakke ;;; Copyright © 2022 Maxime Devos @@ -29,15 +29,16 @@ (define-module (guix git) #:use-module (guix cache) #:use-module (gcrypt hash) #:use-module ((guix build utils) - #:select (mkdir-p delete-file-recursively)) + #:select (mkdir-p delete-file-recursively invoke/quiet)) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix records) #:use-module (guix gexp) #:autoload (guix git-download) (git-reference-url git-reference-commit git-reference-recursive?) + #:autoload (guix config) (%git) #:use-module (guix sets) - #:use-module ((guix diagnostics) #:select (leave warning)) + #:use-module ((guix diagnostics) #:select (leave warning info)) #:use-module (guix progress) #:autoload (guix swh) (swh-download commit-id?) #:use-module (rnrs bytevectors) @@ -428,6 +429,35 @@ (define (delete-checkout directory) (rename-file directory trashed) (delete-file-recursively trashed))) +(define (packs-in-git-repository directory) + "Return the number of pack files under DIRECTORY, a Git checkout." + (catch 'system-error + (lambda () + (let ((directory (opendir (in-vicinity directory ".git/objects/pack")))) + (let loop ((count 0)) + (match (readdir directory) + ((? eof-object?) + (closedir directory) + count) + (str + (loop (if (string-suffix? ".pack" str) + (+ 1 count) + count))))))) + (const 0))) + +(define (maybe-run-git-gc directory) + "Run 'git gc' in DIRECTORY if needed." + ;; XXX: As of libgit2 1.3.x (used by Guile-Git), there's no support for GC. + ;; Each time a checkout is pulled, a new pack is created, which eventually + ;; takes up a lot of space (lots of small, poorly-compressed packs). As a + ;; workaround, shell out to 'git gc' when the number of packs in a + ;; repository has become "too large", potentially wasting a lot of space. + ;; See . + (when (> (packs-in-git-repository directory) 25) + (info (G_ "compressing cached Git repository at '~a'...~%") + directory) + (invoke/quiet %git "-C" directory "gc"))) + (define* (update-cached-checkout url #:key (ref '()) @@ -515,6 +545,9 @@ (define* (update-cached-checkout url seconds seconds nanoseconds nanoseconds)))) + ;; Run 'git gc' if needed. + (maybe-run-git-gc cache-directory) + ;; When CACHE-DIRECTORY is a sub-directory of the default cache ;; directory, remove expired checkouts that are next to it. (let ((parent (dirname cache-directory))) base-commit: 6b0a32196982a0a2f4dbb59d35e55833a5545ac6 -- 2.41.0 From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 30 Oct 2023 12:05:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.169866749931104 (code B ref 66650); Mon, 30 Oct 2023 12:05:02 +0000 Received: (at 66650) by debbugs.gnu.org; 30 Oct 2023 12:04:59 +0000 Received: from localhost ([127.0.0.1]:44442 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qxR0t-00085W-4I for submit@debbugs.gnu.org; Mon, 30 Oct 2023 08:04:59 -0400 Received: from mira.cbaines.net ([212.71.252.8]:42990) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qxR0p-00085H-Nd; Mon, 30 Oct 2023 08:04:56 -0400 Received: from localhost (pd95b13b0.dip0.t-ipconnect.de [217.91.19.176]) by mira.cbaines.net (Postfix) with ESMTPSA id 5E33E27BBE2; Mon, 30 Oct 2023 12:04:21 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 955346f3; Mon, 30 Oct 2023 12:04:18 +0000 (UTC) References: <87jzswsrlt.fsf@gnu.org> User-agent: mu4e 1.10.5; emacs 28.2 From: Christopher Baines Date: Mon, 30 Oct 2023 12:02:59 +0000 In-reply-to: Message-ID: <87sf5swc3j.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Fixes . > > This fixes a bug whereby libgit2-managed checkouts would keep growing as > we fetch. > > * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New > procedures. > (update-cached-checkout): Use it. > --- > guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++--- > 1 file changed, 36 insertions(+), 3 deletions(-) > > Hi! > > This is a radical fix/workaround for the unbounded Git checkout growth > problem, shelling out to =E2=80=98git gc=E2=80=99 when it=E2=80=99s likel= y needed (=E2=80=9Ctoo many=E2=80=9D > pack files around). > > I thought we might be able to implement a =E2=80=98git gc=E2=80=99 approx= imation using > the libgit2 =E2=80=9Cpackbuilder=E2=80=9D interface, but I haven=E2=80=99= t got around to doing > it: . > > Once again, shelling out is not my favorite option, but it=E2=80=99s a bu= g we > should fix sooner rather than later, hence this compromise. > > Thoughts? This sounds good to me, the data service has this problem as well of cached checkouts that grow to be too large and this sounds like it'll address it. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmU/m8BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xcx2A//f3qqeG4+4J5uvVPcP/26SMtpzvNCUsRs 5hzUgJherxg4U25tlUdXlpjSVTNDSc0qN7RpUQWy6rca/S/ro2NL/KcR7VAdyumu v39ldNwbq0W2YC/eZ9fxS7SzeCnWV2oOpO5X5sy69TGvjE/plWNStttvOF/HFy6Y CtH9GNXZE9xwL5PbRK1Lxun5JmamP3Lxk+oivN3ZC7AnhoYxGxJ4xHD9HfN970b/ 6kBbc0Vcf0wHtHRVIEuhBw01JkklchBhTTzbYoi3SUFZeovbkm4Ys5g3s7nDVVZD 5XNMkdp2YcgdUtsfZN1jhgFXTsa6XyfFnQS/1qMfPg3U1niAj0nKIqjEfmUNnGD1 PpQbQ5WvOZm5S70HHDG9Cg58BVIcH0hrHqfVYyghhttf2yUvdKp6CtqNyuBzsr7D 276K8EAeTMcfQwtArxKaFfFG/ggInMvPy1UA1FoN2j0EIIxeND/7vcejqqIssjZm jsU716+s9bP1JCf0s/gJPWSw7Iph7gOs4CKFUdQSeEqNawyXyetxc5PjI6K7NKzq QSa7SJlTe0Lv8maRIZ7LV8t08n3PPFO0sFC7MQMVTCFbkwqFwaVGGdeFiF8dYXfN m4eigk0nl9Poq7gQ79r0igy/rfkZW8mVKRucqVdUJ/znykJfiyBbadSmPcCZP6MS XzvO0Cgwki8= =sesm -----END PGP SIGNATURE----- --=-=-=-- From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 14 Nov 2023 09:20:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Christopher Baines Cc: Josselin Poiret , 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.16999535962936 (code B ref 66650); Tue, 14 Nov 2023 09:20:02 +0000 Received: (at 66650) by debbugs.gnu.org; 14 Nov 2023 09:19:56 +0000 Received: from localhost ([127.0.0.1]:60019 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r2paO-0000lH-23 for submit@debbugs.gnu.org; Tue, 14 Nov 2023 04:19:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:39730) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r2paL-0000kr-HX; Tue, 14 Nov 2023 04:19:54 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r2pZY-0004TY-Te; Tue, 14 Nov 2023 04:19:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=JZDU1hOSj2ITSd6DjETVBVkYN5+QtFH0XZM216c9Cuc=; b=kdEbrjIYkhUUgzl1O0BG z4mkDThAEOWVKdLJu5dwi6F0U0RBk6hj78YdTDQqIERlIJXAW6DRj9+d0Cio1OrcxTvQiAbOdQnPG ret1TpZDdGdeUS0uRurMQqjzddTomVBoMEStUEpTLBDX64ibAgXqimdmMkPnUDsndrKbPiLS43wOi 8JiRM3pHJr1hR1NT762nRI3csilJ3L7OUtI87viVA/P5DaKWZiGlANo9Lz5Cm7gFRAVLqJEZpiwud +l6U3JGnsQiCMFH6CxaGDPeX9ING1cJHWPlzUr/eGPQdsfh5j6dJ0GZspypgY5uktRkbOBxHQkGPY VDKqnJgU+seV6A==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <87sf5swc3j.fsf@cbaines.net> (Christopher Baines's message of "Mon, 30 Oct 2023 12:02:59 +0000") References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> Date: Tue, 14 Nov 2023 10:19:01 +0100 Message-ID: <87o7fwae0q.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Christopher Baines skribis: > Ludovic Court=C3=A8s writes: > >> Fixes . >> >> This fixes a bug whereby libgit2-managed checkouts would keep growing as >> we fetch. [...] > This sounds good to me, the data service has this problem as well of > cached checkouts that grow to be too large and this sounds like it'll > address it. Thanks for your input, Chris. Any other comments? I=E2=80=99d like to push the patch within a few days if there are no objections. https://issues.guix.gnu.org/66650 Ludo=E2=80=99. From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] bug#65720: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Simon Tournier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 14 Nov 2023 12:05:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Christopher Baines Cc: Josselin Poiret , 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.169996346620556 (code B ref 66650); Tue, 14 Nov 2023 12:05:02 +0000 Received: (at 66650) by debbugs.gnu.org; 14 Nov 2023 12:04:26 +0000 Received: from localhost ([127.0.0.1]:60186 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r2s9Z-0005LP-O3 for submit@debbugs.gnu.org; Tue, 14 Nov 2023 07:04:26 -0500 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]:39148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r2s9X-0005Kr-ND; Tue, 14 Nov 2023 07:04:24 -0500 Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-32f831087c6so1028145f8f.0; Tue, 14 Nov 2023 04:03:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699963416; x=1700568216; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=ylTF2Qwpjb3oXQkbeS9e3AnLlKkJ1OUyQRtwgU+E1+Q=; b=AwP+SsOPMQHvKYvI/o4aU2byBlVjQ/CaHhUssWI6cpQiahhd1zCpvl6U2/utawpQBx ZTZvA+Vk2sSqMnvz/OHFVRQsTUHRPNqJ0Iyf7sQVS7FbXNtxnL5f3zmXfSnqv8fw7uRO 9S8lXYSKh66vvSv7jcDKKU4XqDUTUoms6CoyLcIx3xyXhCY2eZ3azAt8b9Y5EtTuLEUz A/zdHK2nJ5BWAxqjUccSJ5LQz3yvypZcd2bZ1pjHB/3Yf9cxz3PW7AdYrhdqIzlBtWTN iX3kqQ9rD8TgASpm7v/AZBgUPeOemaIPBgrEHVwTajcVa0ktjskYEx2SY0CAqPoWjSwc 38iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699963416; x=1700568216; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ylTF2Qwpjb3oXQkbeS9e3AnLlKkJ1OUyQRtwgU+E1+Q=; b=lH24OIockUIOGanmqFgbxrGRc7zttqbK3ESKamxrsLW7PnpTCsdHXI0phtt6zvCLml 8yBP7XfUHD4onbzgiItQW/c7qSXWqsVg4mYvFbVf+GNl5BYpADFPkhwsGx5AgC+L01Iy rCkmsMgmlVhHCZxn1EeZ5fyBNlP88KEXX9n84Na/mKgS/y29/dJeYMQQ+NOyRq1UbwHR inOwBbax1uVPo/df/xqlNEXppNOm2756cw9ZyG+0fvYL6ixJEkhHZBINOuEHdJC88ZBp UO5PPoi8tLHIOxQLUtDpdf6MjbJlb5yWVCH+/JGyTytulOszcRIm6qUqQVwQmotqrNTK dS8w== X-Gm-Message-State: AOJu0YxmDycvu0Kzn756BoKs/IPx6TdQsZTlu7/M2iYMK2IL5k9fZlHf sMvVouVj9jniSrUgVTcBwkM2c3I4ZV4= X-Google-Smtp-Source: AGHT+IGxrX8kNQ98k5MVjas+2EFIinZ9YvkPaQ15MARAjwxnTfVMWW20wlN2YlpNmNhmPTWnsvkRBQ== X-Received: by 2002:a5d:648c:0:b0:331:3cec:214f with SMTP id o12-20020a5d648c000000b003313cec214fmr1209144wri.7.1699963415483; Tue, 14 Nov 2023 04:03:35 -0800 (PST) Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id u30-20020adfa19e000000b00318147fd2d3sm7718993wru.41.2023.11.14.04.03.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Nov 2023 04:03:35 -0800 (PST) From: Simon Tournier In-Reply-To: <87o7fwae0q.fsf@gnu.org> References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> <87o7fwae0q.fsf@gnu.org> Date: Tue, 14 Nov 2023 10:32:41 +0100 Message-ID: <87v8a4el3a.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, On Tue, 14 Nov 2023 at 10:19, Ludovic Court=C3=A8s wrote: > Any other comments? I=E2=80=99d like to push the patch within a few days= if > there are no objections. As mentioned in [1], >> * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New >> procedures. >> (update-cached-checkout): Use it. >> --- >> guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++--- >> 1 file changed, 36 insertions(+), 3 deletions(-) LGTM. Just two colors for the bikeshed. :-) >> + (when (> (packs-in-git-repository directory) 25) Why 25? And not 10 or 50 or 100? >> (define* (update-cached-checkout url >> #:key >> (ref '()) >> @@ -515,6 +545,9 @@ (define* (update-cached-checkout url >> seconds seconds >> nanoseconds nanoseconds)))) >>=20=20 >> + ;; Run 'git gc' if needed. >> + (maybe-run-git-gc cache-directory) Why not trigger it by =E2=80=9Cguix gc=E2=80=9D? Well, I expect =E2=80=9Cguix gc=E2=80=9D to take some time and I choose whe= n. However, I want =E2=80=9Cguix pull=E2=80=9D or =E2=80=9Cguix time-machine=E2=80=9D t= o be as fast as possible and here some extra time is added, and I cannot control exactly when. Cheers, simon 1: bug#65720: [PATCH] git: Shell out to =E2=80=98git gc=E2=80=99 when neces= sary. Simon Tournier Mon, 23 Oct 2023 12:08:07 +0200 id:87il6xlkhk.fsf@gmail.com https://issues.guix.gnu.org/65720 https://issues.guix.gnu.org/msgid/87il6xlkhk.fsf@gmail.com https://yhetil.org/guix/87il6xlkhk.fsf@gmail.com From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] bug#65720: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Nov 2023 12:13:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Simon Tournier Cc: Josselin Poiret , Christopher Baines , 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.170013675126757 (code B ref 66650); Thu, 16 Nov 2023 12:13:02 +0000 Received: (at 66650) by debbugs.gnu.org; 16 Nov 2023 12:12:31 +0000 Received: from localhost ([127.0.0.1]:54656 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3bEV-0006xS-HR for submit@debbugs.gnu.org; Thu, 16 Nov 2023 07:12:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:50414) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3bEU-0006x9-HC; Thu, 16 Nov 2023 07:12:30 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r3bEO-0001BF-E9; Thu, 16 Nov 2023 07:12:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=ZEuFNinQlxo+OxQc1igD8XwkMi/joiRI0BI6c1SFqec=; b=ZFjlK3F4OWU+fez3iiWz IxsLp6ZtJRjErJj/o4gtgt1JCY7E1PHVTZ4ZDEmq8mORUCPR4y6rB/DAREMu8mI6i/s/p1QDwfE3w NNmVmuW0BNR6G2inIgfnUCeavacFkv5ww/W9Q2AcbcV1oFsh5VAOgD9AoA20LjlgDhS6jADhiRDrk kdY88EHA5DJsmDNCcPAAVz6qiiCcWG5Q6Ezg8GueQzuKdHm1pBap47ThK44XZK4jbniweKJ6S6JYX UKIptuWX6hh36Zb+mowvorzPnBB7SUWV1hRc3D9Q/u4D5B1oYB3H9qR+XrFKfbu9+wZfef7YUb/uB qbKsTlXJU438Cw==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: <87v8a4el3a.fsf@gmail.com> (Simon Tournier's message of "Tue, 14 Nov 2023 10:32:41 +0100") References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> <87o7fwae0q.fsf@gnu.org> <87v8a4el3a.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Sextidi 26 Brumaire an 232 de la =?UTF-8?Q?R=C3=A9volution,?= jour de la Pistache X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 16 Nov 2023 13:12:22 +0100 Message-ID: <87h6ll28yh.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Simon Tournier skribis: >>> * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New >>> procedures. >>> (update-cached-checkout): Use it. >>> --- >>> guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++--- >>> 1 file changed, 36 insertions(+), 3 deletions(-) > > LGTM. Thanks! > Just two colors for the bikeshed. :-) > > >>> + (when (> (packs-in-git-repository directory) 25) > > Why 25? And not 10 or 50 or 100? Totally arbitrary. :-) I sampled the checkouts I had on my laptop and that seems like a reasonable heuristic. In particular, it seems that Git-managed checkouts never have this many packs; only libgit2-managed checkouts do, precisely because libgit2 doesn=E2=80=99t repack/GC. >>> + ;; Run 'git gc' if needed. >>> + (maybe-run-git-gc cache-directory) > > Why not trigger it by =E2=80=9Cguix gc=E2=80=9D? Because so far the idea is that ~/.cache/guix/checkouts is automatically managed without user intervention; it=E2=80=99s really a cache in that sens= e. > Well, I expect =E2=80=9Cguix gc=E2=80=9D to take some time and I choose w= hen. However, > I want =E2=80=9Cguix pull=E2=80=9D or =E2=80=9Cguix time-machine=E2=80=9D= to be as fast as possible and > here some extra time is added, and I cannot control exactly when. Yes, I see. The thing is =E2=80=98maybe-run-git-gc=E2=80=99 is only called= on the slow path; so for example, it=E2=80=99s not called on a =E2=80=98time-machine=E2= =80=99 cache hit, but only on a cache miss, which is already expensive anyway. Does that make sense? Thanks, Ludo=E2=80=99. From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] bug#65720: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Simon Tournier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Nov 2023 13:26:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , Christopher Baines , 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.170014111012882 (code B ref 66650); Thu, 16 Nov 2023 13:26:02 +0000 Received: (at 66650) by debbugs.gnu.org; 16 Nov 2023 13:25:10 +0000 Received: from localhost ([127.0.0.1]:54777 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3cMo-0003Lb-5Z for submit@debbugs.gnu.org; Thu, 16 Nov 2023 08:25:10 -0500 Received: from mail-vk1-xa32.google.com ([2607:f8b0:4864:20::a32]:41402) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r3cMm-0003LE-4P; Thu, 16 Nov 2023 08:25:09 -0500 Received: by mail-vk1-xa32.google.com with SMTP id 71dfb90a1353d-4abf066fa12so164595e0c.1; Thu, 16 Nov 2023 05:25:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700141103; x=1700745903; darn=debbugs.gnu.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=iT1oPeGFA7W1gIoMlp6My9hhZFQktk1eIxQtFZfDzbg=; b=Ed4hVnf0O55lV44Z9GY3Rhfpwgg/x23hJrRdeBg5mr1sIhK1fz9yM8RgeBDxFUq4D2 wXDNRqlb2BGJ41kaAtyzOWEWb/lVKPf8KCT6VRkxl5ew29kynlle159Y4hxgUYLsYq1A ZiRxaHANTDrZaj2Ncvi7wn2Ogql2He7f3aJ8cDvjA96vhiCKZRBBcCsGDZsdR8kUheHH apZNU8vr5Bl591napvUQ4WcfTDfIYcTXKSzqV4toZdbGayWYL2F8p/3KZvujLQlCyiTk cCpkX/rAjVBOufaxJRfBFL2nlH3eAZZDyJFiIyzVOLEyoemh6CSBHDNLvR0MjPPYp31A K8Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700141103; x=1700745903; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iT1oPeGFA7W1gIoMlp6My9hhZFQktk1eIxQtFZfDzbg=; b=affSzyL/wkSJFlt/npcguRlsap6WAEwtWz95L0cikApyA21gC9/tMNZTsiKzuhIwWo BKmlaz+8CAG+km1ucV6g9X+tjhyumwCYUuUHLeylg7lZfJ0UZQqxxl3qRXcI4bJuhPxr ePNK3VVX3azwqzsFHzBUYg8MKjHmX/rzzBa4pHhsKYWRROlC8Xon5YZEZF4SsMCQvhWP N+99DfE6gL5PZj6w03gQ0ljW9i3pnwuO1CqxxPDKh6aJ4PRMHQHSTlm1r0S2Ru9dV84e 4r4oc2NTlZgQzeyf+msvfjBiNH1FHOF9hkMezesUgo8Ggp90bPJSvs+yV32/GtvbmzNy 4/HQ== X-Gm-Message-State: AOJu0YxcyJxcEoSO9np/VcosNj8L+rn3dKmP3VxwEffwU5BVQjb2Tb5K gXxBlkmPn71ofWU+nCrWdYehxqFyZ81TLla4gyk= X-Google-Smtp-Source: AGHT+IERgt2ppQkKfWIZOIy7XSrP92IpnO9osFJyZ726t8riBobcc325dQ1HELzZa60bxTifVgexAjqSZNDmoWScr8g= X-Received: by 2002:a05:6102:2c02:b0:460:f279:ae0e with SMTP id ie2-20020a0561022c0200b00460f279ae0emr8825695vsb.1.1700141102724; Thu, 16 Nov 2023 05:25:02 -0800 (PST) MIME-Version: 1.0 References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> <87o7fwae0q.fsf@gnu.org> <87v8a4el3a.fsf@gmail.com> <87h6ll28yh.fsf@gnu.org> In-Reply-To: <87h6ll28yh.fsf@gnu.org> From: Simon Tournier Date: Thu, 16 Nov 2023 14:24:51 +0100 Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, On Thu, 16 Nov 2023 at 13:12, Ludovic Court=C3=A8s wrote: > > Well, I expect =E2=80=9Cguix gc=E2=80=9D to take some time and I choose= when. However, > > I want =E2=80=9Cguix pull=E2=80=9D or =E2=80=9Cguix time-machine=E2=80= =9D to be as fast as possible and > > here some extra time is added, and I cannot control exactly when. > > Yes, I see. The thing is =E2=80=98maybe-run-git-gc=E2=80=99 is only call= ed on the slow > path; so for example, it=E2=80=99s not called on a =E2=80=98time-machine= =E2=80=99 cache hit, but > only on a cache miss, which is already expensive anyway. What you mean as "only called on the slow path" is each time 'update-cached-checkout' is called, right? So, somehow when 'maybe-run-git-gc' is called appears to me "unpredictable". But anyway. :-) Let move it elsewhere if I am really annoyed. Cheers, simon From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] bug#65720: [bug#66650] [PATCH] git: Shell out to =?UTF-8?Q?=E2=80=98git_?= =?UTF-8?Q?gc=E2=80=99?= when necessary. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 22 Nov 2023 11:19:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Simon Tournier Cc: Josselin Poiret , Christopher Baines , 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.17006518907325 (code B ref 66650); Wed, 22 Nov 2023 11:19:02 +0000 Received: (at 66650) by debbugs.gnu.org; 22 Nov 2023 11:18:10 +0000 Received: from localhost ([127.0.0.1]:58124 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r5lFC-0001u3-2u for submit@debbugs.gnu.org; Wed, 22 Nov 2023 06:18:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:34650) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r5lFA-0001tn-1U; Wed, 22 Nov 2023 06:18:08 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r5lF1-000317-3G; Wed, 22 Nov 2023 06:17:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=MWXC4TYqLrnQXs1ngFX4mcOjA6MZowvoq66b2g7zDa4=; b=gF9gJtmSMNyKdkRRimV/ dTSlbTvLr6f0VYtMS92CkyrCM7diHPtITK475CkSmNnAmirZ9LGIeYYdDyXEQAcbGNLu+BS4JV6m6 WcBDswGI0iJ4BOjKd+UE22FPLLR0pe/H5UQW46kdiyyZE5n69FEUDn8RL7mNTlNUXh634I4icFBVM QCOJYklIXJt283yHPrjdkABtUzjE5JYXgPpd7bbYm//3ihyVxaRqVekexHIQ9X3hy7SNU9ANuhehh sSicMTKJigaLvwV3++y7oCsHPEoGpRfOaNDncctftsdVigWlbSzknQkL4332zOieKctvlX9ZRa0Ha QQU+NzvHAmMzUQ==; From: Ludovic =?UTF-8?Q?Court=C3=A8s?= In-Reply-To: (Simon Tournier's message of "Thu, 16 Nov 2023 14:24:51 +0100") References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> <87o7fwae0q.fsf@gnu.org> <87v8a4el3a.fsf@gmail.com> <87h6ll28yh.fsf@gnu.org> Date: Wed, 22 Nov 2023 12:17:55 +0100 Message-ID: <874jhem3z0.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Simon Tournier skribis: > On Thu, 16 Nov 2023 at 13:12, Ludovic Court=C3=A8s wrote: > >> > Well, I expect =E2=80=9Cguix gc=E2=80=9D to take some time and I choos= e when. However, >> > I want =E2=80=9Cguix pull=E2=80=9D or =E2=80=9Cguix time-machine=E2=80= =9D to be as fast as possible and >> > here some extra time is added, and I cannot control exactly when. >> >> Yes, I see. The thing is =E2=80=98maybe-run-git-gc=E2=80=99 is only cal= led on the slow >> path; so for example, it=E2=80=99s not called on a =E2=80=98time-machine= =E2=80=99 cache hit, but >> only on a cache miss, which is already expensive anyway. > > What you mean as "only called on the slow path" is each time > 'update-cached-checkout' is called, right? Yes, which usually indicates we=E2=80=99re on a cache miss (for example a c= ache miss of =E2=80=98guix time-machine=E2=80=99) and thus are going to do poten= tially more work (updating a Git repo, building things, etc.). That=E2=80=99s why I th= ink it=E2=80=99s on the =E2=80=9Cslow path=E2=80=9D and shouldn=E2=80=99t make = much of a difference. More importantly, unless I=E2=80=99m mistaken, it=E2=80=99s rarely going to fire. > So, somehow when 'maybe-run-git-gc' is called appears to me > "unpredictable". But anyway. :-) Sure, but the way I see it, that=E2=80=99s the nature of caches. > Let move it elsewhere if I am really annoyed. :-/ Ludo=E2=80=99. From unknown Sun Jun 22 04:07:19 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#66650] bug#65720: Guile-Git-managed checkouts grow way too much Resent-From: Simon Tournier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 22 Nov 2023 11:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66650 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , Christopher Baines , 65720@debbugs.gnu.org, 66650@debbugs.gnu.org Received: via spool by 66650-submit@debbugs.gnu.org id=B66650.170065425111821 (code B ref 66650); Wed, 22 Nov 2023 11:58:02 +0000 Received: (at 66650) by debbugs.gnu.org; 22 Nov 2023 11:57:31 +0000 Received: from localhost ([127.0.0.1]:58135 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r5lrG-00034Z-GA for submit@debbugs.gnu.org; Wed, 22 Nov 2023 06:57:30 -0500 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]:40975) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r5lrD-00034I-By; Wed, 22 Nov 2023 06:57:28 -0500 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-408363c2918so7060895e9.0; Wed, 22 Nov 2023 03:57:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700654238; x=1701259038; darn=debbugs.gnu.org; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=dOZ3jbw+h+F/Dnxm5Tc+po224qPHyu2O+nU8on3+UEM=; b=LhLi/VFwMxItHl7cBd8eRK0y8/70OB5G4/zixHzY/wy1+jqXK7TpSkq0nZuVC232Y4 JGVagwBZMH/0ymMYNdG73cnsOa0j4sqTyEMuTG4NIcpDakj+0FAVumzxZEbgMeZ6Smhb wVaohdJNUqWT6h9r4ThdS1n131jxaCVPQzhnfoXTj9C3vmPwO1ZtlRvT49q1GtG2RVBC bV/SX+xvuAbc0PIarXRLOEzgllpy19r41ywCW/rr0m0PIHdJ1ZBENGjD+ddU0ZdaLUs+ SP98BeA/tdYAUXc2UN4b+4KGqUcLJDXu1SMGEZIpsZaUhxiRVnGasBZm8/CH7jBwhada iKww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700654238; x=1701259038; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=dOZ3jbw+h+F/Dnxm5Tc+po224qPHyu2O+nU8on3+UEM=; b=ERKCa1i/12vc+ugaHO6N2w3fbBduts60Dk1w+kt3C4eVY76G4bd9QTR2ZO/01J39CZ CNRaouwv0Xv/J6+w1KA8BCOSbN71MdKso3gPhMRiLuj6U1ubC+ZJGQ4b8aAuNIekTyi6 hzyQpEbFUdKejH2OmLDbQ3VwUrGD9Rl+uhnaTBHUeouMJymPgb5rDIuiwfpguUB9Iw0M rmhs3J+hE0cIowC9LpThWBdziJwvpBrwo+zTk1Zre+WS7OaZVGbIMcHIXuS2MYlS87Nb fioZXldJhl0LQPn+7NUurd1aT+MkmGOhAUDcofmA78J2pQTYO4WcmAEFjQYTQoLEB8XW sb3w== X-Gm-Message-State: AOJu0YzJUf3yJq0aLcoVuC1omT9KGYqmtci8MxxTiG+cnvAbr5gIUqFi f5ECQ5cNrH4CLtgYQbaINboXKbwsE3U= X-Google-Smtp-Source: AGHT+IFlRPsJJM81tcYy8kyNkmToAEmfISRsI1/v/tKT29cOZh6jY+BRhKq87dh9hC9IReMHZqKt9w== X-Received: by 2002:a05:600c:3104:b0:40b:2aaf:e066 with SMTP id g4-20020a05600c310400b0040b2aafe066mr1638948wmo.2.1700654238007; Wed, 22 Nov 2023 03:57:18 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id s21-20020a05600c45d500b0040a48430837sm2000668wmo.13.2023.11.22.03.57.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Nov 2023 03:57:17 -0800 (PST) From: Simon Tournier References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> <87o7fwae0q.fsf@gnu.org> <87v8a4el3a.fsf@gmail.com> <87h6ll28yh.fsf@gnu.org> <874jhem3z0.fsf@gnu.org> Date: Wed, 22 Nov 2023 12:57:15 +0100 In-Reply-To: <874jhem3z0.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Wed, 22 Nov 2023 12:17:55 +0100") Message-ID: <86ttpehug4.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, Thanks for explaining. On Wed, 22 Nov 2023 at 12:17, Ludovic Court=C3=A8s wrote: > it=E2=80=99s rarely going to fire. [...] >> Let move it elsewhere if I am really annoyed. > > :-/ Sorry, I poorly worded my last comment. :-) Somehow I was expressing: my view probably falls into the =E2=80=9CPremature optimization is the root of all evil=E2=80=9D category. Other said, I have= no objection and I will revisit the issue when I will be on fire, if I am, or annoyed for real. Cheers, simon PS: Aside this patch: >> So, somehow when 'maybe-run-git-gc' is called appears to me >> "unpredictable". But anyway. :-) > > Sure, but the way I see it, that=E2=80=99s the nature of caches. What makes cache unpredictable is their current state. However, this does not imply that *all* the actions modifying from one state to another must also be triggered in unpredictable moment. For instance, I choose when I wash family=E2=80=99s clothes and the wash-ma= chine does not start by itself when the unpredictable stack of family=E2=80=99s d= irty clothes is enough. Because, maybe today it=E2=80=99s rainy so drying is difficult and tomorrow will be sunny so it will be a better moment. :-) For me, =E2=80=9Cguix gc=E2=80=9D should be the driver for cleaning all the= various Guix caches. Anyway. :-D From unknown Sun Jun 22 04:07:19 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Subject: bug#66650: closed (Re: [bug#66650] bug#65720: Guile-Git-managed checkouts grow way too much) Message-ID: References: <87h6ldkcc8.fsf@gnu.org> X-Gnu-PR-Message: they-closed 66650 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 66650@debbugs.gnu.org Date: Wed, 22 Nov 2023 16:01:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1700668862-26744-1" This is a multi-part message in MIME format... ------------=_1700668862-26744-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #66650: [PATCH] git: Shell out to =E2=80=98git gc=E2=80=99 when necessary. which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 66650@debbugs.gnu.org. --=20 66650: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D66650 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1700668862-26744-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 66650-done) by debbugs.gnu.org; 22 Nov 2023 16:00:37 +0000 Received: from localhost ([127.0.0.1]:59708 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r5peX-0006w1-72 for submit@debbugs.gnu.org; Wed, 22 Nov 2023 11:00:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:43556) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1r5peW-0006ve-9Q; Wed, 22 Nov 2023 11:00:36 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r5peM-0008H9-WA; Wed, 22 Nov 2023 11:00:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=gOeBvHBjrYk1lK6NZsKFKkZxBvOCYRIce4ZSgF8qBSw=; b=GvxX8mE6Uo1o2+3E9xyr aWfc3kwEgxI6nfGYOTlKEnORwhg1HBpyR8Lo5YRKbC9lWAk16oKhmfRgmHbBRv35JW4UMz66rs/f+ yFZ8ss4qCFXlMO54jYDRGHNaJaHKbYlKH7ATY1GGxBg5DHO33vBhZ38IVnYxt7EInE2Ec28oP2bKx uU6GYmEiFxVIsDAyG1PdFR0LZqhxZSZpI3yK61yCs5BsMhUu5aAz7Vy9aXpoCi8QjzflP+bDJYbQq wnHLpPZMc8t10EwT7b37CZlqDeEvUI2LBJvW7DmiO8by61f4HHfjE7gSH3ADH1kwgyVYYBEAZFVSZ hIiZYWIhjrw3qA==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Simon Tournier Subject: Re: [bug#66650] bug#65720: Guile-Git-managed checkouts grow way too much In-Reply-To: <86ttpehug4.fsf_-_@gmail.com> (Simon Tournier's message of "Wed, 22 Nov 2023 12:57:15 +0100") References: <87jzswsrlt.fsf@gnu.org> <87sf5swc3j.fsf@cbaines.net> <87o7fwae0q.fsf@gnu.org> <87v8a4el3a.fsf@gmail.com> <87h6ll28yh.fsf@gnu.org> <874jhem3z0.fsf@gnu.org> <86ttpehug4.fsf_-_@gmail.com> Date: Wed, 22 Nov 2023 17:00:07 +0100 Message-ID: <87h6ldkcc8.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 66650-done Cc: Josselin Poiret , Christopher Baines , 65720-done@debbugs.gnu.org, 66650-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Simon Tournier skribis: > Somehow I was expressing: my view probably falls into the =E2=80=9CPremat= ure > optimization is the root of all evil=E2=80=9D category. Other said, I ha= ve no > objection and I will revisit the issue when I will be on fire, if I am, > or annoyed for real. Alright! Pushed as b150c546b04c9ebb09de9f2c39789221054f5eea. Let=E2=80=99s see how it behaves and if there are problems we had overlooke= d=E2=80=A6 Ludo=E2=80=99. ------------=_1700668862-26744-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 20 Oct 2023 16:16:13 +0000 Received: from localhost ([127.0.0.1]:41305 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qtsAW-0004fs-VW for submit@debbugs.gnu.org; Fri, 20 Oct 2023 12:16:13 -0400 Received: from lists.gnu.org ([2001:470:142::17]:49828) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qtsAV-0004f0-F5 for submit@debbugs.gnu.org; Fri, 20 Oct 2023 12:16:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qts9t-0008NU-Dy for guix-patches@gnu.org; Fri, 20 Oct 2023 12:15:33 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qts9r-0004Rf-PD; Fri, 20 Oct 2023 12:15:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=guykA3Jje4B/fdh8d1YKTVE8F+9iXXIl3FUKarfygvY=; b=jF0iDL/GbSiYleynVOTG /8r/H3EkXts9gzZHBgd9R8nNRQDQE9k6G5S40965qhyaaKSTNmvZmEP05W378rPNL287CkVM+Ibc7 7E32AmsdJM3Am6miqMoPX9YkU5o+Q5z+orHS8Uflfk+Q6GcnHJFgV54Bg6XcAw2I1w6n8BhwLD8XH Xc6+syD3NXx+2uTgwx7sgIiSqx8Q8TyYiO+e+VCDOrgg9PMq1m8eYZ6M32W5h/fNQYw3CoJwQsjTq N7T/JL+bxKLs4dAkgwLr8izbta894kjMEC5GTNjZ2vBkHDJJ991iz+BbSwyZrUfYucrv//2aKtBun iam1wsZbcvK/hA==; From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH] =?UTF-8?q?git:=20Shell=20out=20to=20=E2=80=98git=20gc?= =?UTF-8?q?=E2=80=99=20when=20necessary.?= Date: Fri, 20 Oct 2023 18:15:12 +0200 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: <87jzswsrlt.fsf@gnu.org> References: <87jzswsrlt.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Debbugs-Cc: Christopher Baines , Josselin Poiret , Ludovic Courtès , Mathieu Othacehe , Ricardo Wurmus , Simon Tournier , Tobias Geerinckx-Rice Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= , 65720@debbugs.gnu.org, Josselin Poiret , Simon Tournier X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Fixes . This fixes a bug whereby libgit2-managed checkouts would keep growing as we fetch. * guix/git.scm (packs-in-git-repository, maybe-run-git-gc): New procedures. (update-cached-checkout): Use it. --- guix/git.scm | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) Hi! This is a radical fix/workaround for the unbounded Git checkout growth problem, shelling out to ‘git gc’ when it’s likely needed (“too many” pack files around). I thought we might be able to implement a ‘git gc’ approximation using the libgit2 “packbuilder” interface, but I haven’t got around to doing it: . Once again, shelling out is not my favorite option, but it’s a bug we should fix sooner rather than later, hence this compromise. Thoughts? Ludo’. diff --git a/guix/git.scm b/guix/git.scm index b7182305cf..d704b62333 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2020 Mathieu Othacehe -;;; Copyright © 2018-2022 Ludovic Courtès +;;; Copyright © 2018-2023 Ludovic Courtès ;;; Copyright © 2021 Kyle Meyer ;;; Copyright © 2021 Marius Bakke ;;; Copyright © 2022 Maxime Devos @@ -29,15 +29,16 @@ (define-module (guix git) #:use-module (guix cache) #:use-module (gcrypt hash) #:use-module ((guix build utils) - #:select (mkdir-p delete-file-recursively)) + #:select (mkdir-p delete-file-recursively invoke/quiet)) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix records) #:use-module (guix gexp) #:autoload (guix git-download) (git-reference-url git-reference-commit git-reference-recursive?) + #:autoload (guix config) (%git) #:use-module (guix sets) - #:use-module ((guix diagnostics) #:select (leave warning)) + #:use-module ((guix diagnostics) #:select (leave warning info)) #:use-module (guix progress) #:autoload (guix swh) (swh-download commit-id?) #:use-module (rnrs bytevectors) @@ -428,6 +429,35 @@ (define (delete-checkout directory) (rename-file directory trashed) (delete-file-recursively trashed))) +(define (packs-in-git-repository directory) + "Return the number of pack files under DIRECTORY, a Git checkout." + (catch 'system-error + (lambda () + (let ((directory (opendir (in-vicinity directory ".git/objects/pack")))) + (let loop ((count 0)) + (match (readdir directory) + ((? eof-object?) + (closedir directory) + count) + (str + (loop (if (string-suffix? ".pack" str) + (+ 1 count) + count))))))) + (const 0))) + +(define (maybe-run-git-gc directory) + "Run 'git gc' in DIRECTORY if needed." + ;; XXX: As of libgit2 1.3.x (used by Guile-Git), there's no support for GC. + ;; Each time a checkout is pulled, a new pack is created, which eventually + ;; takes up a lot of space (lots of small, poorly-compressed packs). As a + ;; workaround, shell out to 'git gc' when the number of packs in a + ;; repository has become "too large", potentially wasting a lot of space. + ;; See . + (when (> (packs-in-git-repository directory) 25) + (info (G_ "compressing cached Git repository at '~a'...~%") + directory) + (invoke/quiet %git "-C" directory "gc"))) + (define* (update-cached-checkout url #:key (ref '()) @@ -515,6 +545,9 @@ (define* (update-cached-checkout url seconds seconds nanoseconds nanoseconds)))) + ;; Run 'git gc' if needed. + (maybe-run-git-gc cache-directory) + ;; When CACHE-DIRECTORY is a sub-directory of the default cache ;; directory, remove expired checkouts that are next to it. (let ((parent (dirname cache-directory))) base-commit: 6b0a32196982a0a2f4dbb59d35e55833a5545ac6 -- 2.41.0 ------------=_1700668862-26744-1--