GNU bug report logs - #32161
(guix store deduplication) fails with large files

Previous Next

Full log

View this message in rfc822 format

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: 32161 <at> debbugs.gnu.org
Subject: bug#32161: seek out of range
Date: Thu, 19 Jul 2018 18:03:24 +0200

Hi Ricardo,

Ricardo Wurmus <rekado <at> elephly.net> skribis:

> Ricardo Wurmus <rekado <at> elephly.net> writes:
>
>> I’m getting this bug on berlin.guixsd.org.  The store is several hundred
>> GB in size.  I cannot reproduce this on a machine with a smaller store.
>
> This is not correct.  I cannot reproduce this on a machine where
> deduplication has been disabled.
>
>> --8<---------------cut here---------------start------------->8---
>> In guix/store/deduplication.scm:
>>     62:18  1 (nar-sha256 _)
>> In unknown file:
>>            0 (seek #<output: string 11b20e0> 0 1)
>>
>> ERROR: In procedure seek:
>> Value out of range -2147483648 to 2147483647: 4770726968
>> @ hook-failed /gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv - 256 builder for `/gnu/store/qjxwff3fajh350chpswbb6x9q2m4c3sd-texlive-texmf-2017.drv' failed with exit code 1
>> --8<---------------cut here---------------end--------------->8---

For the record, this code can be executed through ‘guix offload’, via
‘restore-file-set’.

> Line 62 is (port-position wrapper).  “seek” takes an integer as the
> offset and the range it reports is that of the minimum and maximum
> values of a 32 bit integer.

I have some good news!  I fixed this in Guile commit
d677aca5c5e5b3a9f71af57243169904ba4a712c.

Bad news, we can’t really work around it on the Guix side.

There’s also another bug: the ‘write’ procedure of a custom binary
output port is never called when passing a bytevector bigger than 2^32.

> So… does this mean that we need some other way of counting written bytes
> to get rid of “counting-wrapper-port”?

Commit 83099892e0cf0d9c59f5e1a0774331026e48baa8 gets rid of it, but
that’s mostly for aesthetic and performance reasons.

> On berlin.guixsd.org I may have to temporarily disable deduplication.

I’m not sure that’d help.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.