GNU bug report logs - #44760
Closure copy in ‘guix system init’ is inefficient

Previous Next

Full log

Message #16 received at 44760 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 44760 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 00/15] Speed up 'guix system init' & co.
Date: Fri, 11 Dec 2020 16:09:10 +0100

Hi there!

Here’s a long and rather boring patch series to address
<https://issues.guix.gnu.org/44760> and a bit more.

To avoid traversing store items repeatedly as described in the
issue above, the strategy here is to gradually move the
reset-timestamps and deduplicate phases as part of the file
copying process, such that each file is accessed only once.
Consequently, the kitchen sink that ‘register-items’ once was
is now very focused.

Furthermore, it changes ‘guix system init’ so that it reuses
the already-known store item hashes when populating the target
database instead of re-traversing store items.

On my laptop (SSD, warm cache, derivations already built), the
command:

  guix system init gnu/system/examples/bare-bones.tmpl /tmp/sys

goes from 32s to 22s, a 33% improvement.

Feedback welcome!

Ludo’.

Ludovic Courtès (15):
  serialization: 'fold-archive' notifies about directory processing
    completion.
  serialization: 'restore-file' sets canonical timestamp and
    permissions.
  nar: Deduplicate files right as they are restored.
  store-copy: 'populate-store' resets timestamps.
  image: 'register-closure' assumes already-reset timestamps.
  database: Remove #:reset-timestamps? from 'register-items'.
  store-copy: 'populate-store' can optionally deduplicate files.
  image: 'register-closure' leaves it up to the caller to deduplicate.
  database: Remove #:deduplicate? from 'register-items'.
  guix system: 'init' copies, resets timestamps, and deduplicates at
    once.
  database: Remove #:deduplicate? and #:reset-timestamps? from
    'register-path'.
  system: 'init' does not recompute the hash of each store item.
  database: Remove 'register-path'.
  database: Honor 'SOURCE_DATE_EPOCH'.
  deduplicate: Create the '.links' directory lazily.

 .dir-locals.el                |   1 +
 gnu/build/image.scm           |  16 +-
 gnu/build/install.scm         |   3 +-
 gnu/build/linux-initrd.scm    |   3 +-
 gnu/build/vm.scm              |  14 +-
 gnu/system/install.scm        |  12 +-
 gnu/system/linux-initrd.scm   |  10 +-
 guix/build/store-copy.scm     | 133 ++++++++++++----
 guix/nar.scm                  |   8 +-
 guix/scripts/archive.scm      |   2 +
 guix/scripts/challenge.scm    |   1 +
 guix/scripts/pack.scm         | 276 +++++++++++++++++-----------------
 guix/scripts/system.scm       |  64 ++++----
 guix/serialization.scm        |  36 +++--
 guix/store/database.scm       |  58 ++-----
 guix/store/deduplication.scm  | 167 ++++++++++++++------
 tests/gexp.scm                |  20 ++-
 tests/guix-archive.sh         |   4 +-
 tests/nar.scm                 |  21 ++-
 tests/store-database.scm      |  18 ++-
 tests/store-deduplication.scm |  20 ++-
 21 files changed, 544 insertions(+), 343 deletions(-)

-- 
2.29.2

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.