GNU bug report logs - #51787
GC takes more than 9 hours on berlin

Previous Next

Full log

Message #100 received at 51787 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 51787 <at> debbugs.gnu.org
Subject: Re: Disk performance on ci.guix.gnu.org
Date: Sat, 25 Dec 2021 23:19:23 +0100

Ricardo Wurmus <rekado <at> elephly.net> writes:

> Today we discovered a few more things and discussed them on IRC.  Here’s
> a summary.
>
> /var/cache sits on the same storage as /gnu.  We mounted the 5TB ext4
> file system that’s hosted by the SAN at /mnt_test and started copying
> over /var/cache to /mnt_test/var/cache.  Transfer speed was considerably
> faster (not *great*, but reasonably fast) than the copy of
> /gnu/store/trash to the same target.

Turns out that space on the SAN is insufficient for a full copy of
/var/cache.  We’ve hit ENOSPC after 4.2TB.  The SAN enforces some
headroom to remain free, so it denies us full access to the 5TB slice.
Bummer.

I guess we’ll have to wait for the SAN extension some time early 2022
before we can relocate the substitutes cache.

Should we attempt to overwrite /gnu/store and rely exclusively on
substitutes from the cache?

No matter how we look at it, the huge store is a performance problem for
us.  Today I had to kill ’guix gc’ after the GC lock had been held for
about 24 hours.  We will keep having this problem.

-- 
Ricardo

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.