GNU bug report logs - #42001
“SQLite database is busy”: contention on the store database

Previous Next

Full log

Message #11 received at 42001 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 42001 <at> debbugs.gnu.org
Subject: Re: bug#42001: “SQLite database is
 busy”: contention on the store database
Date: Tue, 23 Jun 2020 09:42:02 +0200

Hi,

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

>> utimensat(AT_FDCWD, "/gnu/store/whhc80cx1zpi9cj3bil3sp8wwzcf8p9p-image-root/gnu/store/n6l40ygclxfysnc2nxwrslnn5ab6wv3v-profile/lib/modules/5.4.47-gnu/kernel/net/netfilter", [{tv_sec=1592814617, tv_nsec=396083395} /* 2020-06-22T10:30:17.396083395+0200 */, {tv_sec=1, tv_nsec=0} /* 1970-01-01T01:00:01+0100 */], 0) = 0
>> chmod("/gnu/store/whhc80cx1zpi9cj3bil3sp8wwzcf8p9p-image-root/gnu/store/n6l40ygclxfysnc2nxwrslnn5ab6wv3v-profile/lib/modules/5.4.47-gnu/kernel/net/netfilter", 040555) = 0
>> getrusage(RUSAGE_SELF, {ru_utime={tv_sec=1306, tv_usec=334062}, ru_stime={tv_sec=143, tv_usec=819675}, ...}) = 0
>> clock_gettime(CLOCK_PROCESS_CPUTIME_ID, {tv_sec=1450, tv_nsec=153748295}) = 0 
>> lstat("/gnu/store/whhc80cx1zpi9cj3bil3sp8wwzcf8p9p-image-root/gnu/store/n6l40ygclxfysnc2nxwrslnn5ab6wv3v-profile/lib/modules/5.4.47-gnu/kernel/net/netfilter/xt_connlimit.ko", {st_mode=S_IFLNK|0777, st_size=125, ...}) = 0
>
> As discussed on IRC, those "image-root" derivations do exercise store
> deduplication a lot, are could participate to the observed database
> contention.
>
> Turns out I used those derivations as "staging" directories that are
> passed to GNU Xorriso when producing ISO images and "mke2fs" and
> "mkdosfs" when producing raw disk-images. Using the store was also
> convenient to ensure root ownership of the image files.
>
> With 1cb9effc3e875ec7bae31bb06069d16ac89f7e1d and
> 7f75a7ec08975eb6d6e01db61bd6b91f447f655e, I remove those "image-root"
> derivations, and proceed to image creation in a single derivation.

Nice, thanks for the prompt change!

I’ve been monitoring berlin a bit and I’ve become quite convinced that
these big directories are what triggered contention.  The good thing is
that this has allowed us to find weaknesses in (guix store database) and
(guix store deduplicate).

> The later commit is a bit more tricky because "mke2fs" is not able to
> override file permissions. I used "fakeroot" to make sure that the raw
> disk-image files are owned by "root" and not "guixbuilder".

OK, makes sense.

Maybe at some point we’ll have to use libext2fs directly or patch mke2fs
to get the desired flexibility.

Thank you!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.