Subject: bug#44760: closed (Re: bug#44760: [PATCH 00/15] Speed up 'guix system init' & co.)

From: [email-masked] (GNU bug Tracking System)
Date: Tue, 15 Dec 2020 16:35:01 +0000

Your bug report

#44760: Closure copy in ‘guix system init’ is inefficient

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
44760: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=44760
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#44760: [PATCH 00/15] Speed up 'guix system init' & co.
From: Ludovic Courtès <[email-masked]>
To: [email-masked]
Date: Tue, 15 Dec 2020 17:33:57 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

Ludovic Courtès <[email-masked]> skribis:

>   serialization: 'fold-archive' notifies about directory processing
>     completion.
>   serialization: 'restore-file' sets canonical timestamp and
>     permissions.
>   nar: Deduplicate files right as they are restored.
>   store-copy: 'populate-store' resets timestamps.
>   image: 'register-closure' assumes already-reset timestamps.
>   database: Remove #:reset-timestamps? from 'register-items'.
>   store-copy: 'populate-store' can optionally deduplicate files.
>   image: 'register-closure' leaves it up to the caller to deduplicate.
>   database: Remove #:deduplicate? from 'register-items'.
>   guix system: 'init' copies, resets timestamps, and deduplicates at
>     once.
>   database: Remove #:deduplicate? and #:reset-timestamps? from
>     'register-path'.
>   system: 'init' does not recompute the hash of each store item.
>   database: Remove 'register-path'.
>   database: Honor 'SOURCE_DATE_EPOCH'.
>   deduplicate: Create the '.links' directory lazily.

Pushed as 7530e491b517497b7b8166b5ccecdc3d4cdb468d!

Ludo'.

Subject: Closure copy in ‘guix system init’ is inefficient
From: Ludovic Courtès <[email-masked]>
To: <[email-masked]>
Date: Fri, 20 Nov 2020 12:02:25 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

‘guix system init’ ends by copying the system’s closure from the “host”
store to the target store; it also initializes the database of that
target store.

That copy is inefficient for several reasons.  Let’s pick one file,
shred.1.gz, that ends up being copied, and let’s look at its occurrences
in the strace log of ‘guix system init config.scm /tmp/os’:

--8<---------------cut here---------------start------------->8---
$ grep -A2 '/shred.1.gz' ,,s
lstat("/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", {st_mode=S_IFREG|0444, st_size=1490, ...}) = 0
openat(AT_FDCWD, "/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", O_RDONLY) = 15
fstat(15, {st_mode=S_IFREG|0444, st_size=1490, ...}) = 0
openat(AT_FDCWD, "/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", O_WRONLY|O_CREAT|O_TRUNC, 0444) = 16
read(15, "\37\213\10\0\0\0\0\0\2\3\215VMs\3336\20\275\363Wluh\354\251L%vg\322:M"..., 8192) = 1490
write(16, "\37\213\10\0\0\0\0\0\2\3\215VMs\3336\20\275\363Wluh\354\251L%vg\322:M"..., 1490) = 1490
--
utimensat(AT_FDCWD, "/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", [{tv_sec=1605721025, tv_nsec=616985411} /* 2020-11-18T18:37:05.616985411+0100 */, {tv_sec=1, tv_nsec=0} /* 1970-01-01T01:00:01+0100 */], 0) = 0
lstat("/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/sleep.1.gz", {st_mode=S_IFREG|0444, st_size=813, ...}) = 0
openat(AT_FDCWD, "/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/sleep.1.gz", O_RDONLY) = 15
--
lstat("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", {st_mode=S_IFREG|0444, st_size=1490, ...}) = 0
lstat("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shuf.1.gz", {st_mode=S_IFREG|0444, st_size=972, ...}) = 0
lstat("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/sleep.1.gz", {st_mode=S_IFREG|0444, st_size=813, ...}) = 0
--
lstat("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", {st_mode=S_IFREG|0444, st_size=1490, ...}) = 0
openat(AT_FDCWD, "/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", O_RDONLY) = 17
lseek(17, 0, SEEK_CUR)                  = 0
read(17, "\37\213\10\0\0\0\0\0\2\3\215VMs\3336\20\275\363Wluh\354\251L%vg\322:M"..., 1490) = 1490
--
lstat("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", {st_mode=S_IFREG|0444, st_size=1490, ...}) = 0
openat(AT_FDCWD, "/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", O_RDONLY) = 17
lseek(17, 0, SEEK_CUR)                  = 0
read(17, "\37\213\10\0\0\0\0\0\2\3\215VMs\3336\20\275\363Wluh\354\251L%vg\322:M"..., 1490) = 1490
--
link("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shred.1.gz", "/tmp/os/gnu/store/.links/0w0qcs5lp36i89yry91r2ixlghihzf0vc56bpd9yylj342gv82xl") = 0
lstat("/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shuf.1.gz", {st_mode=S_IFREG|0444, st_size=972, ...}) = 0
openat(AT_FDCWD, "/tmp/os/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/share/man/man1/shuf.1.gz", O_RDONLY) = 17
--8<---------------cut here---------------end--------------->8---

First, /tmp/os/…/shred.1.gz is read entirely twice: once in
‘register-items’ (in the ‘nar-sha256’ call) to compute its hash, and a
second time for deduplication (the ‘deduplicate’ call in there.)

The ‘nar-sha256’ call could be avoided because the database of
/gnu/store contains that value.  As for deduplication, we could perhaps
create those ‘.links’ entries as we copy files instead of re-traversing
the whole thing afterwards.

Second, all of /tmp/os is traversed to reset timestamps, although we
could have cleared those timestamps when we created those files in the
first place (<https://issues.guix.gnu.org/44741> prevents that though,
unless we keep a bug-fixed copy of ‘copy-recursively’ in there.)

Third, in the case of the installer, we’re really copying from
/mnt/guix-inst/store to /mnt/gnu/store, which is likely the same
device.  In this case we could create hard links instead of actually
copying files.

Fourth, we’re adding items one by one in the target store database, but
it may be more efficient to more or less dump the subset of the source
database in bulk.

Surely we can do better.

Ludo’.