GNU bug report logs - #23666
'add-to-store' RPC loads files in memory (daemon-side)

Previous Next

To reply to this bug, email your comments to 23666 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: bug-guix <at> gnu.org
Subject: guix download fails for large files
Date: Tue, 31 May 2016 21:56:31 +0200

Hello,

since I am regularly rebuilding texlive (who is not!), I would like to keep
the source tarball on my machine, and whenever I garbage collect it from the
store, add it back via
   guix download file:texlive-20150523-texmf.tar.xz

On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
   guix download: error: sendfile: Broken pipe

The file itself has 1,8 GB.

Downloading the file via http as part of "guix build" works, and
"guix download file:" also works on my x86_64 machine with 8 GB of RAM.

Andreas

Message #8 received at 23666 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Andreas Enge <andreas <at> enge.fr>
Cc: 23666 <at> debbugs.gnu.org
Subject: Re: bug#23666: guix download fails for large files
Date: Tue, 31 May 2016 16:26:51 -0400

On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote:
> since I am regularly rebuilding texlive (who is not!), I would like to keep
> the source tarball on my machine, and whenever I garbage collect it from the
> store, add it back via
>    guix download file:texlive-20150523-texmf.tar.xz
>
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
>    guix download: error: sendfile: Broken pipe
> 
> The file itself has 1,8 GB.
> 
> Downloading the file via http as part of "guix build" works, and
> "guix download file:" also works on my x86_64 machine with 8 GB of RAM.

As a temporary workaround, you could make the tarball a gcroot:

# ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \
/var/guix/gcroots/texlive-texmf.tar.xz

Message #11 received at 23666 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Andreas Enge <andreas <at> enge.fr>
Cc: 23666 <at> debbugs.gnu.org
Subject: Re: bug#23666: guix download fails for large files
Date: Wed, 01 Jun 2016 14:39:54 +0200

Andreas Enge <andreas <at> enge.fr> skribis:

> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
>    guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.

This is implemented using the ‘add-to-store’ RPC, which, after all these
years, is still implemented like this (nix/libstore/local-store.cc):

--8<---------------cut here---------------start------------->8---
Path LocalStore::addToStore(const string & name, const Path & _srcPath,
    bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
{
    Path srcPath(absPath(_srcPath));
    debug(format("adding `%1%' to the store") % srcPath);

    /* Read the whole path into memory. This is not a very scalable
       method for very large paths, but `copyPath' is mainly used for
       small files. */
--8<---------------cut here---------------end--------------->8---

This is undoubtedly what’s causing the failure.

Until this is fixed, you could wrap the file in a nar and import the nar
(nar import does not have this limitation, see ’LocalStore::importPath’
and co.).

To do that, you could do:

  guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar

and later:

  guix archive --import < t.nar

HTH,
Ludo’.

Message #16 received at 23666 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 23666 <at> debbugs.gnu.org
Subject: Re: bug#23666: guix download fails for large files
Date: Sun, 5 Jun 2016 22:58:39 +0200

Thanks for your suggestions, Leo and Ludovic! I still see this as a bug;
should I report it upstream to Nix?

On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
> This is implemented using the ‘add-to-store’ RPC, which, after all these
> years, is still implemented like this (nix/libstore/local-store.cc):
> 
> --8<---------------cut here---------------start------------->8---
> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>     bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
> {
>     Path srcPath(absPath(_srcPath));
>     debug(format("adding `%1%' to the store") % srcPath);
> 
>     /* Read the whole path into memory. This is not a very scalable
>        method for very large paths, but `copyPath' is mainly used for
>        small files. */
> --8<---------------cut here---------------end--------------->8---

Something that mainly does not fail could indeed be seen as a bug...
But how come that "guix download http://" succeeds, where
"guix download file://" fails?

Andreas

Message #19 received at 23666 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Andreas Enge <andreas <at> enge.fr>
Cc: 23666 <at> debbugs.gnu.org
Subject: Re: bug#23666: guix download fails for large files
Date: Mon, 06 Jun 2016 09:06:07 +0200

Andreas Enge <andreas <at> enge.fr> skribis:

> On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
>> This is implemented using the ‘add-to-store’ RPC, which, after all these
>> years, is still implemented like this (nix/libstore/local-store.cc):
>> 
>> --8<---------------cut here---------------start------------->8---
>> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>>     bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
>> {
>>     Path srcPath(absPath(_srcPath));
>>     debug(format("adding `%1%' to the store") % srcPath);
>> 
>>     /* Read the whole path into memory. This is not a very scalable
>>        method for very large paths, but `copyPath' is mainly used for
>>        small files. */
>> --8<---------------cut here---------------end--------------->8---
>
> Something that mainly does not fail could indeed be seen as a bug...
> But how come that "guix download http://" succeeds, where
> "guix download file://" fails?

Both use the ‘add-to-store’ RPC, so both have equal chances of failing.

Note that, fortunately, fixed-output derivations (‘guix build -S foo’)
do not use ‘add-to-store’.  In general, except in the case of ‘guix
download’, ‘add-to-store’ is only used with small files like when adding
.scm files to the store, etc.

Ludo’.

Message #26 received at 23666-done <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: 23666-done <at> debbugs.gnu.org, 30706-done <at> debbugs.gnu.org,
 43274-done <at> debbugs.gnu.org, 67048-done <at> debbugs.gnu.org
Subject: Close
Date: Sun, 6 Jul 2025 22:20:09 +0200

Closing old issues.

Andreas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.