GNU bug report logs - #71357
[PATCH] substitute: Don’t keep cache entries more than a few days.

Previous Next

Full log

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH] substitute: Don’t keep cache entries more than a few days.
Date: Tue,  4 Jun 2024 10:06:43 +0200

Experience has shown that keeping too many entries increases disk usage
and, more importantly, leads to long delays when cleaning up the cache,
measured in minutes on slow or busy HDDs with hundreds of thousands of
cache entries, as is common on build machines.  In those cases, the cost
of the cache outweighs its benefit.

* guix/scripts/substitute.scm (%narinfo-expired-cache-entry-removal-delay):
Reduce to 5 days.
(cached-narinfo-expiration-time)[max-ttl]: Reduce to 2 days.

Change-Id: Iab212f572ee9041be61716423a3c014f93fe81ed
---
 guix/scripts/substitute.scm | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

Hello,

Chris mentioned it before and I experienced it the hard way on bayfront:

  https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00177.html

A big narinfo cache is a significant performance hit on spinning HDDs
when the time comes to remove expired entries.

This change makes the cache more ephemeral (2 to 5 days).  I still think
some caching is needed: one will often run several Guix commands in a
day that will query the same narinfos and will only download/build a
small subset (keep in mind that that ‘substitution-oracle’, used by
‘derivation-build-plan’, query narinfos for the closure of the requested
derivations, minus those already valid); it would be wasteful and
inefficient to download them over and over again.  I’d like to have
metrics to estimate that, but I don’t.

Thoughts?

Ludo’.

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index a7ad56dbcd5..8bcbca5e7aa 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -77,7 +77,7 @@ (define-module (guix scripts substitute)
 
 (define %narinfo-expired-cache-entry-removal-delay
   ;; How often we want to remove files corresponding to expired cache entries.
-  (* 7 24 3600))
+  (* 5 24 3600))
 
 (define (warn-about-missing-authentication)
   (warning (G_ "authentication and authorization of substitutes \
@@ -169,8 +169,9 @@ (define (cached-narinfo-expiration-time file)
   "Return the expiration time for FILE, which is a cached narinfo."
   (define max-ttl
     ;; Upper bound on the TTL used to avoid keeping around cached narinfos for
-    ;; too long, which makes the cache bigger and more expensive to traverse.
-    (* 2 30 24 60 60))                            ;2 months
+    ;; too long, which makes the cache bigger and more expensive to traverse
+    ;; when deleting old entries.
+    (* 2 24 60 60))
 
   (catch 'system-error
     (lambda ()

base-commit: 85ac164c41fc4c93d3cb2a5d3321c63598c2855f
-- 
2.45.1

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.