GNU bug report logs - #23666
'add-to-store' RPC loads files in memory (daemon-side)

Previous Next

Full log

View this message in rfc822 format

From: ludo <at> gnu.org (Ludovic Courtès)
To: Andreas Enge <andreas <at> enge.fr>
Cc: 23666 <at> debbugs.gnu.org
Subject: bug#23666: guix download fails for large files
Date: Wed, 01 Jun 2016 14:39:54 +0200

Andreas Enge <andreas <at> enge.fr> skribis:

> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
>    guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.

This is implemented using the ‘add-to-store’ RPC, which, after all these
years, is still implemented like this (nix/libstore/local-store.cc):

--8<---------------cut here---------------start------------->8---
Path LocalStore::addToStore(const string & name, const Path & _srcPath,
    bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
{
    Path srcPath(absPath(_srcPath));
    debug(format("adding `%1%' to the store") % srcPath);

    /* Read the whole path into memory. This is not a very scalable
       method for very large paths, but `copyPath' is mainly used for
       small files. */
--8<---------------cut here---------------end--------------->8---

This is undoubtedly what’s causing the failure.

Until this is fixed, you could wrap the file in a nar and import the nar
(nar import does not have this limitation, see ’LocalStore::importPath’
and co.).

To do that, you could do:

  guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar

and later:

  guix archive --import < t.nar

HTH,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.