From debbugs-submit-bounces@debbugs.gnu.org Sat Dec 19 17:06:40 2020 Received: (at submit) by debbugs.gnu.org; 19 Dec 2020 22:06:40 +0000 Received: from localhost ([127.0.0.1]:43329 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqkN5-0000HG-Lg for submit@debbugs.gnu.org; Sat, 19 Dec 2020 17:06:39 -0500 Received: from lists.gnu.org ([209.51.188.17]:57348) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqkN3-0000H3-Hn for submit@debbugs.gnu.org; Sat, 19 Dec 2020 17:06:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41782) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kqkN3-0007iq-Ce for guix-patches@gnu.org; Sat, 19 Dec 2020 17:06:37 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:35424) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kqkN3-0002bw-3e; Sat, 19 Dec 2020 17:06:37 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41858 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kqkN2-0006yy-7i; Sat, 19 Dec 2020 17:06:36 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH] git: Periodically delete least-recently-used cached checkouts. Date: Sat, 19 Dec 2020 23:06:30 +0100 Message-Id: <20201219220630.24605-1-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This ensures ~/.cache/guix/checkouts is periodically cleaned up. * guix/git.scm (cached-checkout-expiration) (%checkout-cache-cleanup-period): New variables. (delete-checkout): New procedure. (update-cached-checkout)[cache-entries]: New procedure. Add call to 'maybe-remove-expired-cache-entries'. --- guix/git.scm | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) Hi! I noticed that my ~/.cache/guix/checkouts directory had accumulated a lot of cruft from channels, playing with ‘--with-branch’ and such, and that it would be nice to clean it up once in a while. This is what this patch does. It uses the (guix cache) default strategy, which consists in deleting least-recently-used items by looking at their atime. Thoughts? Ludo’. diff --git a/guix/git.scm b/guix/git.scm index ca77b9f54b..5df11db38e 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -23,8 +23,10 @@ #:use-module (git submodule) #:use-module (guix i18n) #:use-module (guix base32) + #:use-module (guix cache) #:use-module (gcrypt hash) - #:use-module ((guix build utils) #:select (mkdir-p)) + #:use-module ((guix build utils) + #:select (mkdir-p delete-file-recursively)) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix records) @@ -35,6 +37,7 @@ #:use-module (rnrs bytevectors) #:use-module (ice-9 format) #:use-module (ice-9 match) + #:use-module (ice-9 ftw) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-34) @@ -318,6 +321,20 @@ definitely available in REPOSITORY, false otherwise." (_ #f))) +(define cached-checkout-expiration + ;; Return the expiration time of a cached checkout. + (file-expiration-time (* 30 24 3600))) + +(define %checkout-cache-cleanup-period + ;; Period for the removal of expired cached checkouts. + (* 5 24 3600)) + +(define (delete-checkout directory) + "Delete DIRECTORY recursively, in an atomic fashion." + (let ((trashed (string-append directory ".trashed"))) + (rename-file directory trashed) + (delete-file-recursively trashed))) + (define* (update-cached-checkout url #:key (ref '(branch . "master")) @@ -341,6 +358,14 @@ When RECURSIVE? is true, check out submodules as well, if any. When CHECK-OUT? is true, reset the cached working tree to REF; otherwise leave it unchanged." + (define (cache-entries directory) + (filter-map (match-lambda + ((or "." "..") + #f) + (file + (string-append directory "/" file))) + (or (scandir directory) '()))) + (define canonical-ref ;; We used to require callers to specify "origin/" for each branch, which ;; made little sense since the cache should be transparent to them. So @@ -387,6 +412,17 @@ it unchanged." ;; REPOSITORY as soon as possible. (repository-close! repository) + ;; When CACHE-DIRECTORY is a sub-directory of the default cache + ;; directory, remove expired checkouts that are next to it. + (let ((parent (dirname cache-directory))) + (when (string=? parent (%repository-cache-directory)) + (maybe-remove-expired-cache-entries parent cache-entries + #:entry-expiration + cached-checkout-expiration + #:delete-entry delete-checkout + #:cleanup-period + %checkout-cache-cleanup-period))) + (values cache-directory (oid->string oid) relation))))) (define* (latest-repository-commit store url -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Sun Dec 20 05:46:52 2020 Received: (at 45327) by debbugs.gnu.org; 20 Dec 2020 10:46:52 +0000 Received: from localhost ([127.0.0.1]:43759 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqwEl-0006O2-Re for submit@debbugs.gnu.org; Sun, 20 Dec 2020 05:46:52 -0500 Received: from mout02.posteo.de ([185.67.36.66]:51275) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqwEj-0006Np-Ai for 45327@debbugs.gnu.org; Sun, 20 Dec 2020 05:46:50 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout02.posteo.de (Postfix) with ESMTPS id 0F58B2400FF for <45327@debbugs.gnu.org>; Sun, 20 Dec 2020 11:46:42 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1608461203; bh=hgAGqq5Ux3iOk/3JECpocMbU9Cze7GjtVnnswfgpjog=; h=From:To:Cc:Subject:Date:From; b=gkbGzoXmVLEiBrwYd4QZ26IhKRJvkcfi40HXdhg5DI73ItqRAiNiGR05iMzatvdAi gfagb0MBNchh+IHsHj0L19EzJjDmZrwQ52zPEePU1Kx+OFo610U1ua3cAacuiSwmZ0 b+y5a6p3t5Y5ko59yGgGpCaYbyeBZ1fvpazHfPkKC5542sXMZKLoRcjrP7B1RUAnPY b7gmvxTL0stRRAYsIAFV3pD9H2xW2+dIjQIbOfoTddSCCOzg/1rr2xXf+VJFUvnNSw jSI6Y93gEfe/tvo0cJThuBqaAtVPzxUgku5Mo5Yf78Pxcs7s9T1Dfu3PH07rWuqkPC kFa9bM3ceFdLg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4CzK686Jd3z9rxP; Sun, 20 Dec 2020 11:46:40 +0100 (CET) References: <20201219220630.24605-1-ludo@gnu.org> User-agent: mu4e 1.4.13; emacs 27.1 From: Guillaume Le Vaillant To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. In-reply-to: <20201219220630.24605-1-ludo@gnu.org> Date: Sun, 20 Dec 2020 11:46:36 +0100 Message-ID: <87pn34n4ab.fsf@yamatai> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327 Cc: 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s skribis: > Hi! > > I noticed that my ~/.cache/guix/checkouts directory had accumulated > a lot of cruft from channels, playing with =E2=80=98--with-branch=E2=80= =99 and such, > and that it would be nice to clean it up once in a while. > > This is what this patch does. It uses the (guix cache) default > strategy, which consists in deleting least-recently-used items by > looking at their atime. > > Thoughts? How does it behave when the cache is on a file system mounted with the 'noatime' option? --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX98rjA8cZ2x2QHBvc3Rl by5uZXQACgkQa+ggit8h/j9AiQD/Sxhqiead7d7oinH1G5Kz4LpLKWpZ1ZBcToqb Epjn8NsBAISRLoyuznD0wThB6QHJwbW9HJI3nYvBQ6I1CNChzv+4 =3Ed5 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Dec 20 08:47:23 2020 Received: (at 45327) by debbugs.gnu.org; 20 Dec 2020 13:47:23 +0000 Received: from localhost ([127.0.0.1]:43882 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqz3T-0004WI-1O for submit@debbugs.gnu.org; Sun, 20 Dec 2020 08:47:23 -0500 Received: from eggs.gnu.org ([209.51.188.92]:51176) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqz3R-0004W7-SI for 45327@debbugs.gnu.org; Sun, 20 Dec 2020 08:47:22 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46468) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kqz3M-0006PO-Ab; Sun, 20 Dec 2020 08:47:16 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=34152 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kqz3L-0002Yi-B8; Sun, 20 Dec 2020 08:47:16 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Guillaume Le Vaillant Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. References: <20201219220630.24605-1-ludo@gnu.org> <87pn34n4ab.fsf@yamatai> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Frimaire an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 20 Dec 2020 14:47:13 +0100 In-Reply-To: <87pn34n4ab.fsf@yamatai> (Guillaume Le Vaillant's message of "Sun, 20 Dec 2020 11:46:36 +0100") Message-ID: <87eejk8u8u.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327 Cc: 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Guillaume Le Vaillant skribis: > Ludovic Court=C3=A8s skribis: > >> Hi! >> >> I noticed that my ~/.cache/guix/checkouts directory had accumulated >> a lot of cruft from channels, playing with =E2=80=98--with-branch=E2=80= =99 and such, >> and that it would be nice to clean it up once in a while. >> >> This is what this patch does. It uses the (guix cache) default >> strategy, which consists in deleting least-recently-used items by >> looking at their atime. >> >> Thoughts? > > How does it behave when the cache is on a file system mounted with the > 'noatime' option? I guess the worst that could happen is that checkouts are removed too frequently (because the atime is not updated), meaning that users find themselves making full clones more often than we=E2=80=99d like. Perhaps we could use the mtime instead, since when checkouts are updated, the mtime is presumably updated too. Thoughts? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sun Dec 20 09:16:34 2020 Received: (at 45327) by debbugs.gnu.org; 20 Dec 2020 14:16:35 +0000 Received: from localhost ([127.0.0.1]:43941 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqzVi-0005Ed-Ks for submit@debbugs.gnu.org; Sun, 20 Dec 2020 09:16:34 -0500 Received: from mout01.posteo.de ([185.67.36.65]:49215) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kqzVg-0005EN-Th for 45327@debbugs.gnu.org; Sun, 20 Dec 2020 09:16:33 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 67F13160063 for <45327@debbugs.gnu.org>; Sun, 20 Dec 2020 15:16:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1608473786; bh=p+IiPmUXxoIl922Y/kfvQ4rcws9zlQCCZQ9MIXka4AI=; h=From:To:Cc:Subject:Date:From; b=Merq4Ibe83iQTEAO50zjNPTvfY9r44pF4oI5zVpXs6EX5+AWvTD/yTrro/uRDv1Gs ow/1ZwIa6lomAFwwxGl3efISuuqOkwdNHeWvWpmUG+WEb5sTF6THj6tlUI5AdXDmXa sxzwC/ZVC/FFwqtaWFiQhYvP7zneSubB88FB9IxnXEP4fOG/vEkkSoxTThCj1wjmOh 6hxPMcCvMDz6/IioZR6/zjW2vqAD/3iA6bUQLgADpg0gD32yY0XZsIWKB+acBdzR/D MfqBNTHeBKaICuMaKBg3yBTeqgEoMmVho+tDrrDxHhBBquBtSNCRMNwZJoDc6SSVhz 0bxBk7Gpqf2uQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4CzPm93m8Bz6tmB; Sun, 20 Dec 2020 15:16:25 +0100 (CET) References: <20201219220630.24605-1-ludo@gnu.org> <87pn34n4ab.fsf@yamatai> <87eejk8u8u.fsf@gnu.org> User-agent: mu4e 1.4.13; emacs 27.1 From: Guillaume Le Vaillant To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. In-reply-to: <87eejk8u8u.fsf@gnu.org> Date: Sun, 20 Dec 2020 15:16:22 +0100 Message-ID: <87mty8mukp.fsf@yamatai> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327 Cc: 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s skribis: > Hi, > > Guillaume Le Vaillant skribis: > >> Ludovic Court=C3=A8s skribis: >> >>> Hi! >>> >>> I noticed that my ~/.cache/guix/checkouts directory had accumulated >>> a lot of cruft from channels, playing with =E2=80=98--with-branch=E2=80= =99 and such, >>> and that it would be nice to clean it up once in a while. >>> >>> This is what this patch does. It uses the (guix cache) default >>> strategy, which consists in deleting least-recently-used items by >>> looking at their atime. >>> >>> Thoughts? >> >> How does it behave when the cache is on a file system mounted with the >> 'noatime' option? > > I guess the worst that could happen is that checkouts are removed too > frequently (because the atime is not updated), meaning that users find > themselves making full clones more often than we=E2=80=99d like. > > Perhaps we could use the mtime instead, since when checkouts are > updated, the mtime is presumably updated too. > > Thoughts? > > Ludo=E2=80=99. I guess either using mtime or making Guix update the atime when using a cached checkout would work. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX99ctg8cZ2x2QHBvc3Rl by5uZXQACgkQa+ggit8h/j8WfwD/f/8ohoVqsuunW5+oz4nWVry826NaW4eAAeCN ymejEx8BAJiv/7/OyGR0WyDJXLa+79K6VkKbSz4LLZAm7Aj9dgG8 =A+Nf -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 21 05:31:31 2020 Received: (at 45327) by debbugs.gnu.org; 21 Dec 2020 10:31:31 +0000 Received: from localhost ([127.0.0.1]:46387 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1krITT-0000Fd-D5 for submit@debbugs.gnu.org; Mon, 21 Dec 2020 05:31:31 -0500 Received: from mail-wr1-f41.google.com ([209.85.221.41]:42109) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1krITR-0000F8-Nr for 45327@debbugs.gnu.org; Mon, 21 Dec 2020 05:31:30 -0500 Received: by mail-wr1-f41.google.com with SMTP id m5so10488903wrx.9 for <45327@debbugs.gnu.org>; Mon, 21 Dec 2020 02:31:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:in-reply-to:references:date:message-id:mime-version :content-transfer-encoding; bh=+9TSyUgyOjz3qGyCyOWmAMimH2KtRHQUkBjUMr2HRBY=; b=lQkzUvuNf6uFFrxFECsmR++b/jlD+lAKiwGTgks2SqlynM2vTbofQMcczNXBO9/rKf BTwgMk0YwZJSz3OTmiOJghIdsDK81rgyaLXCHclo2LuECc/TrZ+RsAVQNSFKmEpf2jVG gWBd3hUWChzMqHjWcK6sXRQcE3yLCuPiA0K80p83i0SPC7gJAOghQycwI7W8KWQcfleI CmAjtn/IVF1S1XN4Usi/5RbesQ7sWI9DsZnrWpMxCaSsk7A34NGGaC0Lf/ovcRjXJTrb JuYW1Y+If00scTSRmjO9SvP1QGvLQD7rc4IMuEXsJlzj5ZN9TX3UM6USp+gEX+5jFkzw 2V7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=+9TSyUgyOjz3qGyCyOWmAMimH2KtRHQUkBjUMr2HRBY=; b=X0SU3Fgfd3fEfUGSTGlTkicv8VDBQ3RmDZLHIRZ/vreSiFuYhdAU9I3CUsrEDqqOAM YGBJta7dV1spUzM7PqzHwoqbu7ZjLncQKCbJvcz5y27gW3QftHyGghIyiidZ0eQOFP/H xYifpQtSsvBFW5YaWx28qRNka1tjRJ8m95TNgTX4rYs2T4j7xLHCgWhGp2TQtJNGzD5u +Xpu+2NneH7a/599PfG7+aXCgAEdwkwp2Z9D8tMvntWyyJY4HE7okVRnbuFoqreO8M1R EJNoOstPV8XdYh0bp2nk0dt0Umqfl+Rjnl2fm50GuLea6wm4ZgwhcI5oI0km6iMASqfQ WeHQ== X-Gm-Message-State: AOAM532IkyZ0wU6Fir4Ykwj5TeAlxz30FPYt52ZkwxFLNcCiCo52K4PJ 8rb5t5n6Jisi/R6HMJ/r1kD+LadN5kQ= X-Google-Smtp-Source: ABdhPJx98PdarIJhNNUY+WnaX9dVd3wj4X5G7q8it7Yuekl6EO6Bwa4IiTGS0x+egzRlFuJvKGvRaw== X-Received: by 2002:a5d:6a05:: with SMTP id m5mr17684253wru.96.1608546683783; Mon, 21 Dec 2020 02:31:23 -0800 (PST) Received: from lili (oul69-1-82-232-2-3.fbx.proxad.net. [82.232.2.3]) by smtp.gmail.com with ESMTPSA id z63sm6167698wme.8.2020.12.21.02.31.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Dec 2020 02:31:23 -0800 (PST) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= , 45327@debbugs.gnu.org Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. In-Reply-To: <20201219220630.24605-1-ludo@gnu.org> References: <20201219220630.24605-1-ludo@gnu.org> Date: Mon, 21 Dec 2020 11:26:05 +0100 Message-ID: <86a6u7bgle.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45327 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, On Sat, 19 Dec 2020 at 23:06, Ludovic Court=C3=A8s wrote: > This ensures ~/.cache/guix/checkouts is periodically cleaned up. > > * guix/git.scm (cached-checkout-expiration) > (%checkout-cache-cleanup-period): New variables. > (delete-checkout): New procedure. > (update-cached-checkout)[cache-entries]: New procedure. > Add call to 'maybe-remove-expired-cache-entries'. > --- > guix/git.scm | 38 +++++++++++++++++++++++++++++++++++++- > 1 file changed, 37 insertions(+), 1 deletion(-) > > Hi! > > I noticed that my ~/.cache/guix/checkouts directory had accumulated > a lot of cruft from channels, playing with =E2=80=98--with-branch=E2=80= =99 and such, > and that it would be nice to clean it up once in a while. > > This is what this patch does. It uses the (guix cache) default > strategy, which consists in deleting least-recently-used items by > looking at their atime. This is done at pull time, right? Personally, I would prefer at gc time, and even maybe with an option to =E2=80=9Cguix gc=E2=80=9D. Because, IIUC, every 5 days, the entries older than 1 month will be deleted. As an extensive user of the time-machine, it means that I will do this extra work more than often, slowing down the already slow =E2=80=9Ctime-machine=E2=80=9D. Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 22 08:33:47 2020 Received: (at 45327) by debbugs.gnu.org; 22 Dec 2020 13:33:48 +0000 Received: from localhost ([127.0.0.1]:49224 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1krhnP-0004Yk-K8 for submit@debbugs.gnu.org; Tue, 22 Dec 2020 08:33:47 -0500 Received: from eggs.gnu.org ([209.51.188.92]:47838) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1krhnM-0004YU-TQ for 45327@debbugs.gnu.org; Tue, 22 Dec 2020 08:33:46 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58397) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1krhnH-00063Z-Lf; Tue, 22 Dec 2020 08:33:39 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=43826 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1krhnD-0007Hy-FD; Tue, 22 Dec 2020 08:33:36 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. References: <20201219220630.24605-1-ludo@gnu.org> <86a6u7bgle.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 2 =?utf-8?Q?Niv=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 22 Dec 2020 14:33:33 +0100 In-Reply-To: <86a6u7bgle.fsf@gmail.com> (zimoun's message of "Mon, 21 Dec 2020 11:26:05 +0100") Message-ID: <87blem2ceq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327 Cc: 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, zimoun skribis: > On Sat, 19 Dec 2020 at 23:06, Ludovic Court=C3=A8s wrote: [...] >> I noticed that my ~/.cache/guix/checkouts directory had accumulated >> a lot of cruft from channels, playing with =E2=80=98--with-branch=E2=80= =99 and such, >> and that it would be nice to clean it up once in a while. >> >> This is what this patch does. It uses the (guix cache) default >> strategy, which consists in deleting least-recently-used items by >> looking at their atime. > > This is done at pull time, right? This is happens when =E2=80=98update-cached-checkout=E2=80=99 is called: wh= en updating a channel, using =E2=80=98--with-git-url=E2=80=99, etc. > Personally, I would prefer at gc time, and even maybe with an option > to =E2=80=9Cguix gc=E2=80=9D. Hmm yes (currently the two things are unrelated.) I have a slight preference for something automated that you don=E2=80=99t have to worry abo= ut. > Because, IIUC, every 5 days, the entries older than 1 month will be > deleted. Correct. > As an extensive user of the time-machine, it means that I will do this > extra work more than often, slowing down the already slow > =E2=80=9Ctime-machine=E2=80=9D. Let=E2=80=99s say there=E2=80=99s a couple of channels you regularly pull f= rom, like a few times per months. Their checkout would never be deleted so you wouldn=E2=80=99t notice any change in terms of performance. The difference you=E2=80=99d see is if you pull from a few channels, but le= ss than once per month. In that case, the =E2=80=98guix=E2=80=99 channel woul= d remain in cache (because cache cleanup happens after the cached checkout has been updated), but the other channels would be deleted just to be cloned again soon after that. Does that make sense? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 22 10:29:33 2020 Received: (at 45327) by debbugs.gnu.org; 22 Dec 2020 15:29:33 +0000 Received: from localhost ([127.0.0.1]:50725 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1krjbQ-0001lT-Jl for submit@debbugs.gnu.org; Tue, 22 Dec 2020 10:29:33 -0500 Received: from mail-wr1-f51.google.com ([209.85.221.51]:43179) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1krjbO-0001kx-5e for 45327@debbugs.gnu.org; Tue, 22 Dec 2020 10:29:31 -0500 Received: by mail-wr1-f51.google.com with SMTP id y17so14889908wrr.10 for <45327@debbugs.gnu.org>; Tue, 22 Dec 2020 07:29:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=l+P+0NzSXPGUudbAS7i5n1FFvQlBSzzscTF4m4gmFEA=; b=WwAMZvPfGQVDUaBalGUMpcP0s05l7Qvc1YS7uBY0BUI04Ww/w7i04IU4+zi87utDP/ tgQay7ZW9KUWwdxwdZX8yQm6Ak6x4DdGwJUeutLaXeJ/8nhQ/lczUvLhFmvB3JosMAc8 cvH/icbsVRSkGnTKjvhFMl/r1XhCBfSPGkfp4uXCWvs/CHXML136A6d1VaLUmLKZD9Sb SMqJ23yuLxNU/3biNVTrYOUMTcNGhYFiI5dn+TqXPNOTW4iVZy1ogiXSDfMwEzjEjoVT b5BODp+tcNfvnTYH5aIbUnUvQq69xAyRHdEtTzzfoVrZ/8BSUjxdWTYmnTvgPGxBvLs1 OMug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=l+P+0NzSXPGUudbAS7i5n1FFvQlBSzzscTF4m4gmFEA=; b=owVLhrTxKJ//mcmXD4sRYLUMVPsZNHmOmeVMUi68MSh6xDv/ltZ4Fmyac2BA3rrQWn snzJV2u3AWG6Lgbk6iWokq/PbcIncx6gDlSnQhZ4uta5EUzu3es3fQ6glHscM4mddcFq 7HFF0IdbaLpN052p3455/S2/Kyak9jpbZ1LTKYIihMZEQbEcsBjp8GFhWYpDHdHH28go t41S5oypycCIXCsJMgfq0dKbprRn19gXqIWoKGk1CKD4Ch9RaeZdFUhm+mIEFinzdUXV udEuncRaP+uF4nxbFbnTQGWXFZM705cIeJqZ578UaqvLQCMFpXW5FkUyscK0USqbMcIS e9SQ== X-Gm-Message-State: AOAM531dWme+SkjaQh/q1irEFY2JD9lIRWUzhytP7E0R8Wu8dEOquhg7 XRPg9de1+TI3CicWaxDW7eeb1NVFEUw= X-Google-Smtp-Source: ABdhPJxR+eGuTus2k+eIGMoW9YrnJi43Ng/CMU0BoPoaufG2yjG8WKcBkKtstAMTmQ4hT2xVlj3EwA== X-Received: by 2002:a5d:4491:: with SMTP id j17mr24302759wrq.78.1608650963840; Tue, 22 Dec 2020 07:29:23 -0800 (PST) Received: from lili (oul69-1-82-232-2-3.fbx.proxad.net. [82.232.2.3]) by smtp.gmail.com with ESMTPSA id h14sm31720828wrx.37.2020.12.22.07.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Dec 2020 07:29:23 -0800 (PST) From: zimoun To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. In-Reply-To: <87blem2ceq.fsf@gnu.org> References: <20201219220630.24605-1-ludo@gnu.org> <86a6u7bgle.fsf@gmail.com> <87blem2ceq.fsf@gnu.org> Date: Tue, 22 Dec 2020 16:19:29 +0100 Message-ID: <86wnx97try.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45327 Cc: 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, On Tue, 22 Dec 2020 at 14:33, Ludovic Court=C3=A8s wrote: > The difference you=E2=80=99d see is if you pull from a few channels, but = less > than once per month. In that case, the =E2=80=98guix=E2=80=99 channel wo= uld remain in > cache (because cache cleanup happens after the cached checkout has been > updated), but the other channels would be deleted just to be cloned > again soon after that. > > Does that make sense? All make sense. Thanks for explaining. As said, my preference of such thing is =E2=80=9Cguix gc=E2=80=9D and not = =E2=80=9Cguix pull=E2=80=9D. But since =E2=80=9Cguix pull --delete-generations=E2=80=9D is already here= =E2=80=A6 it pulls by deleting. :-) Anyway, it is =E2=80=9Cguix pull=E2=80=9D. However, I still do not like the =E2=80=9Cautomatic=E2=80=9D part. Persona= lly, I would prefer something like: guix pull --delete-generations=3D2m deleting the cache older than 2 months, in addition to the old profiles. Because, with your patch, if I want to change the expiration or the period, it is not convenient: via channels.scm, maybe. Sometime, I am one or two months off (vacation). And I am sure to forget to tweak the channels.scm when I am back. Well, I have 3-4 channels.scm files; for example I am not pulling guix-past each time I pull. Therefore, I should have these 3-4 channels.scm duplicated with the expiration+period tweaked for when I am back from long breaks. Well, if the automatic is the default, a way to turn off could be nice, at the CLI level or even globally. IMHO. Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 07 04:39:14 2021 Received: (at 45327) by debbugs.gnu.org; 7 Jan 2021 09:39:14 +0000 Received: from localhost ([127.0.0.1]:46447 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxRlC-0004rJ-IA for submit@debbugs.gnu.org; Thu, 07 Jan 2021 04:39:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36734) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxRlB-0004r5-4E for 45327@debbugs.gnu.org; Thu, 07 Jan 2021 04:39:13 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52853) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kxRl4-0002rU-BN; Thu, 07 Jan 2021 04:39:07 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44106 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kxRl3-0000MV-Po; Thu, 07 Jan 2021 04:39:06 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Re: [bug#45327] [PATCH] git: Periodically delete least-recently-used cached checkouts. References: <20201219220630.24605-1-ludo@gnu.org> <86a6u7bgle.fsf@gmail.com> <87blem2ceq.fsf@gnu.org> <86wnx97try.fsf@gmail.com> Date: Thu, 07 Jan 2021 10:39:03 +0100 In-Reply-To: <86wnx97try.fsf@gmail.com> (zimoun's message of "Tue, 22 Dec 2020 16:19:29 +0100") Message-ID: <87v9c9hyrs.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327 Cc: 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, zimoun skribis: > However, I still do not like the =E2=80=9Cautomatic=E2=80=9D part. Perso= nally, I would > prefer something like: > > guix pull --delete-generations=3D2m > > deleting the cache older than 2 months, in addition to the old profiles. > Because, with your patch, if I want to change the expiration or the > period, it is not convenient: via channels.scm, maybe. There=E2=80=99s a difference though: generations are things you manage by yourself as a user, whereas this cache is really just an internal cache. Of course it becomes a user-visible feature if it gets in the way, and I agree we should avoid that. But it also gets in the way by not being vacuumed, hence this patch. > Sometime, I am one or two months off (vacation). And I am sure to > forget to tweak the channels.scm when I am back. Well, I have 3-4 > channels.scm files; for example I am not pulling guix-past each time I > pull. Therefore, I should have these 3-4 channels.scm duplicated with > the expiration+period tweaked for when I am back from long breaks. > > Well, if the automatic is the default, a way to turn off could be nice, > at the CLI level or even globally. IMHO. Yeah, maybe we can have a more conservative default together with an environment variable to tweak it. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 07 05:10:11 2021 Received: (at 45327) by debbugs.gnu.org; 7 Jan 2021 10:10:11 +0000 Received: from localhost ([127.0.0.1]:46534 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxSF8-0005es-S5 for submit@debbugs.gnu.org; Thu, 07 Jan 2021 05:10:11 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44480) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxSF5-0005e1-QA for 45327@debbugs.gnu.org; Thu, 07 Jan 2021 05:10:08 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:53119) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kxSEz-0000Lh-Sh; Thu, 07 Jan 2021 05:10:01 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=44162 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kxSEx-0000hJ-Dm; Thu, 07 Jan 2021 05:10:00 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 45327@debbugs.gnu.org Subject: [PATCH v2] git: Periodically delete least-recently-used cached checkouts. Date: Thu, 7 Jan 2021 11:09:47 +0100 Message-Id: <20210107100947.31189-1-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <86wnx97try.fsf@gmail.com> References: <86wnx97try.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327 Cc: Guillaume Le Vaillant , =?UTF-8?q?Ludovic=20Court=C3=A8s?= , zimoun X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This ensures ~/.cache/guix/checkouts is periodically cleaned up. * guix/git.scm (cached-checkout-expiration) (%checkout-cache-cleanup-period): New variables. (delete-checkout): New procedure. (update-cached-checkout)[cache-entries]: New procedure. Add call to 'maybe-remove-expired-cache-entries'. * guix/cache.scm (file-expiration-time): Add optional 'timestamp' parameter and honor it. --- guix/cache.scm | 9 +++++---- guix/git.scm | 44 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 47 insertions(+), 6 deletions(-) Hi! This v2 changes two things: • Increase the default expiration time to three months; • Check the mtime rather than the atime of checkouts. As zimoun proposed, I agree that we should make the expiration time configurable. I started doing that but that requires ‘string->duration’, which is currently in (guix ui), and this code is supposed to be “UI-free”. So I’d first like to move this and similar tools to a new (guix units) module… Thoughts? Ludo’. diff --git a/guix/cache.scm b/guix/cache.scm index feff131068..0401a9d428 100644 --- a/guix/cache.scm +++ b/guix/cache.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2013, 2014, 2015, 2016, 2017, 2020 Ludovic Courtès +;;; Copyright © 2013, 2014, 2015, 2016, 2017, 2020, 2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -47,13 +47,14 @@ (unless (= ENOENT (system-error-errno args)) (apply throw args))))) -(define (file-expiration-time ttl) +(define* (file-expiration-time ttl #:optional (timestamp stat:atime)) "Return a procedure that, when passed a file, returns its \"expiration -time\" computed as its last-access time + TTL seconds." +time\" computed as its timestamp + TTL seconds. Call TIMESTAMP to obtain the +relevant timestamp from the result of 'stat'." (lambda (file) (match (stat file #f) (#f 0) ;FILE may have been deleted in the meantime - (st (+ (stat:atime st) ttl))))) + (st (+ (timestamp st) ttl))))) (define* (remove-expired-cache-entries entries #:key diff --git a/guix/git.scm b/guix/git.scm index ca77b9f54b..a5103547d3 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -1,6 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2017, 2020 Mathieu Othacehe -;;; Copyright © 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,8 +23,10 @@ #:use-module (git submodule) #:use-module (guix i18n) #:use-module (guix base32) + #:use-module (guix cache) #:use-module (gcrypt hash) - #:use-module ((guix build utils) #:select (mkdir-p)) + #:use-module ((guix build utils) + #:select (mkdir-p delete-file-recursively)) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix records) @@ -35,6 +37,7 @@ #:use-module (rnrs bytevectors) #:use-module (ice-9 format) #:use-module (ice-9 match) + #:use-module (ice-9 ftw) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-34) @@ -318,6 +321,24 @@ definitely available in REPOSITORY, false otherwise." (_ #f))) +(define cached-checkout-expiration + ;; Return the expiration time procedure for a cached checkout. + ;; TODO: Honor $GUIX_GIT_CACHE_EXPIRATION. + + ;; Use the mtime rather than the atime to cope with file systems mounted + ;; with 'noatime'. + (file-expiration-time (* 90 24 3600) stat:mtime)) + +(define %checkout-cache-cleanup-period + ;; Period for the removal of expired cached checkouts. + (* 5 24 3600)) + +(define (delete-checkout directory) + "Delete DIRECTORY recursively, in an atomic fashion." + (let ((trashed (string-append directory ".trashed"))) + (rename-file directory trashed) + (delete-file-recursively trashed))) + (define* (update-cached-checkout url #:key (ref '(branch . "master")) @@ -341,6 +362,14 @@ When RECURSIVE? is true, check out submodules as well, if any. When CHECK-OUT? is true, reset the cached working tree to REF; otherwise leave it unchanged." + (define (cache-entries directory) + (filter-map (match-lambda + ((or "." "..") + #f) + (file + (string-append directory "/" file))) + (or (scandir directory) '()))) + (define canonical-ref ;; We used to require callers to specify "origin/" for each branch, which ;; made little sense since the cache should be transparent to them. So @@ -387,6 +416,17 @@ it unchanged." ;; REPOSITORY as soon as possible. (repository-close! repository) + ;; When CACHE-DIRECTORY is a sub-directory of the default cache + ;; directory, remove expired checkouts that are next to it. + (let ((parent (dirname cache-directory))) + (when (string=? parent (%repository-cache-directory)) + (maybe-remove-expired-cache-entries parent cache-entries + #:entry-expiration + cached-checkout-expiration + #:delete-entry delete-checkout + #:cleanup-period + %checkout-cache-cleanup-period))) + (values cache-directory (oid->string oid) relation))))) (define* (latest-repository-commit store url -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 07 07:40:45 2021 Received: (at 45327) by debbugs.gnu.org; 7 Jan 2021 12:40:45 +0000 Received: from localhost ([127.0.0.1]:46753 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxUar-0003Q4-6k for submit@debbugs.gnu.org; Thu, 07 Jan 2021 07:40:45 -0500 Received: from mail-qt1-f182.google.com ([209.85.160.182]:42171) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kxUap-0003Pp-AA for 45327@debbugs.gnu.org; Thu, 07 Jan 2021 07:40:43 -0500 Received: by mail-qt1-f182.google.com with SMTP id z3so4091453qtw.9 for <45327@debbugs.gnu.org>; Thu, 07 Jan 2021 04:40:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=p6pFltI/BJOOZE5pmZQ0foJVlpKpcB46XXVyJk+zCzk=; b=LKuD2O1N/hh54qpGttoy/3Mt3QLlzow3F5ugcPgXXGLcqn/spFyQ+RSDgy7hfdwgii 5MVu9nGjUPlT2o6w6r8SlW9pxbydtWDLYC2Ee/JSyUL9ZldpGfOeMWu83xdzjrl8bCzH pAf2URW0qwre51SoSewLn6Lps32qVrWnjHaQFX2ccaBYMqCP1AB1Hh+EFKImO1czgQjD CLKrMpjSbrZpfY0cBAXbCsN6Hhx0hwvsdDtC4vb3tjtz4BJwz/17RrIF26Fjh/lyNg+T V3N+IWPeZ1Vm61p1SG/RTg71oLSz4LiFWWQ/wuAkF/J6RBtUlzf2fntix5b3vKMkfhxq gUNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=p6pFltI/BJOOZE5pmZQ0foJVlpKpcB46XXVyJk+zCzk=; b=lhAor4ykdzL86i1tJ3VUaAgryuFt/pPo3SoG4MrBV4Q42NB/cg731fepxPiiKeeVjD Cmsgqg3pb0JEjA1f5bVHns9YQ6WqClbn4watHL7Ne4Qxe32tyrqnaL6hk7NSvkvlQe+m wAAG965fdgW/0c/PkXV9Y1wIkLa0vJLbkEew08X9A9kL3eHhpjkLMnQP2ipoKuHgm7Qi iLGvHMXAFUBPiFbVpGspEWZg/tuBd6oH1OBjKBtfQ7sE4R0jdtmgbu8FwNvG3cGHjFsk SPvGpujXPUqAkfSbhqgMKHzI4TcqWVJO7V7e4LoxCgdNH8BjzwF8MI0tewjSydGlA1Ta AfEA== X-Gm-Message-State: AOAM533IuN33GKM2GjK1T5fG5KiVpWSOKRiWF3pF6+fQQdmh5V2B6akt 4+ZC0FDm1pru88/KzkyWNpm6bLQxHyVElUqsrNw= X-Google-Smtp-Source: ABdhPJxISR/MY4H9Ol/3YMeYUtWOQFjr5fwx79Of1THVmgoTpPjJhAVkoESo4JQt/sgX9Xi27EzjB3rUx3365hfW7qs= X-Received: by 2002:ac8:7141:: with SMTP id h1mr8191082qtp.211.1610023237769; Thu, 07 Jan 2021 04:40:37 -0800 (PST) MIME-Version: 1.0 References: <86wnx97try.fsf@gmail.com> <20210107100947.31189-1-ludo@gnu.org> In-Reply-To: <20210107100947.31189-1-ludo@gnu.org> From: zimoun Date: Thu, 7 Jan 2021 13:40:26 +0100 Message-ID: Subject: Re: [PATCH v2] git: Periodically delete least-recently-used cached checkouts. To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45327 Cc: Guillaume Le Vaillant , 45327@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, On Thu, 7 Jan 2021 at 11:10, Ludovic Court=C3=A8s wrote: > =E2=80=A2 Increase the default expiration time to three months; That's long enough for my vacations. ;-) > As zimoun proposed, I agree that we should make the expiration time > configurable. I started doing that but that requires =E2=80=98string->du= ration=E2=80=99, > which is currently in (guix ui), and this code is supposed to be > =E2=80=9CUI-free=E2=80=9D. So I=E2=80=99d first like to move this and si= milar tools to a new > (guix units) module=E2=80=A6 You mean 2 another patches are coming: one moving to the new (guix units) and then another one honoring GUIX_GIT_CACHE_EXPIRATION, right? Otherwise LGTM. :-) Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 13 10:48:04 2021 Received: (at 45327-done) by debbugs.gnu.org; 13 Jan 2021 15:48:04 +0000 Received: from localhost ([127.0.0.1]:36334 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kziNQ-0000y0-JW for submit@debbugs.gnu.org; Wed, 13 Jan 2021 10:48:04 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53278) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kziNP-0000xU-32 for 45327-done@debbugs.gnu.org; Wed, 13 Jan 2021 10:48:03 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58477) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kziNJ-0006yG-91; Wed, 13 Jan 2021 10:47:57 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39742 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kziNI-0002dq-3O; Wed, 13 Jan 2021 10:47:56 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: zimoun Subject: Re: bug#45327: [PATCH] git: Periodically delete least-recently-used cached checkouts. References: <86wnx97try.fsf@gmail.com> <20210107100947.31189-1-ludo@gnu.org> Date: Wed, 13 Jan 2021 16:47:54 +0100 In-Reply-To: (zimoun's message of "Thu, 7 Jan 2021 13:40:26 +0100") Message-ID: <87wnwg7s9h.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45327-done Cc: Guillaume Le Vaillant , 45327-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, zimoun skribis: > On Thu, 7 Jan 2021 at 11:10, Ludovic Court=C3=A8s wrote: > >> =E2=80=A2 Increase the default expiration time to three months; > > That's long enough for my vacations. ;-) Pffew. :-) Pushed as 87b0001325992db60fdf24ac09ce254cd003721c! >> As zimoun proposed, I agree that we should make the expiration time >> configurable. I started doing that but that requires =E2=80=98string->d= uration=E2=80=99, >> which is currently in (guix ui), and this code is supposed to be >> =E2=80=9CUI-free=E2=80=9D. So I=E2=80=99d first like to move this and s= imilar tools to a new >> (guix units) module=E2=80=A6 > > You mean 2 another patches are coming: one moving to the new (guix > units) and then another one honoring GUIX_GIT_CACHE_EXPIRATION, right? Yup! (Not ETA though=E2=80=A6) Thanks, Ludo=E2=80=99. From unknown Tue Jun 17 01:44:12 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 11 Feb 2021 12:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator