GNU bug report logs - #65033
[PATCH] guix: read-derivation-from-file: Use less open files.

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Christopher Baines <mail <at> cbaines.net>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#65033: closed ([PATCH] guix: read-derivation-from-file: Use
 less open files.)
Date: Tue, 12 Sep 2023 08:34:02 +0000

[Message part 1 (text/plain, inline)]

Your message dated Tue, 12 Sep 2023 09:29:23 +0100
with message-id <87wmwvvli6.fsf <at> cbaines.net>
and subject line Re: [bug#65033] [PATCH] guix: read-derivation-from-file: Use less open files.
has caused the debbugs.gnu.org bug report #65033,
regarding [PATCH] guix: read-derivation-from-file: Use less open files.
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
65033: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=65033
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Christopher Baines <mail <at> cbaines.net>
To: guix-patches <at> gnu.org
Subject: [PATCH] guix: read-derivation-from-file: Use less open files.
Date: Thu,  3 Aug 2023 09:46:12 +0100

The Guix derivation graph isn't that deep, so I don't think this generally
opens lots of files, but I think it's still unnecessary to keep more files
than needed open.

* guix/derivations.scm (read-derivation-from-file): Read each derivation to a
string, which is passed as a port to read-derivation.
---
 guix/derivations.scm | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/guix/derivations.scm b/guix/derivations.scm
index 9fec7f4f0b..2154bd76f6 100644
--- a/guix/derivations.scm
+++ b/guix/derivations.scm
@@ -31,6 +31,7 @@ (define-module (guix derivations)
   #:use-module (ice-9 match)
   #:use-module (ice-9 rdelim)
   #:use-module (ice-9 vlist)
+  #:use-module (ice-9 textual-ports)
   #:use-module (guix store)
   #:use-module (guix utils)
   #:use-module (guix base16)
@@ -556,7 +557,14 @@ (define (read-derivation-from-file file)
   ;; and because the same argument is read more than 15 times on average
   ;; during something like (package-derivation s gdb).
   (or (and file (hash-ref %derivation-cache file))
-      (let ((drv (call-with-input-file file read-derivation)))
+      (let ((drv
+             ;; read-derivation can call read-derivation-from-file, so to
+             ;; avoid having multiple open files when reading a derivation
+             ;; with inputs, read it in to a string first.
+             (call-with-input-string
+                 (call-with-input-file file
+                   get-string-all)
+               read-derivation)))
         (hash-set! %derivation-cache file drv)
         drv)))
 

base-commit: fe3e05d8b3dbb255179d3f85aca870e6085bb71a
prerequisite-patch-id: 2322b3b5ce79bdaa763075cdbb96e760168d4c63
-- 
2.41.0

[Message part 3 (message/rfc822, inline)]

From: Christopher Baines <mail <at> cbaines.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 65033-close <at> debbugs.gnu.org
Subject: Re: [bug#65033] [PATCH] guix: read-derivation-from-file: Use less
 open files.
Date: Tue, 12 Sep 2023 09:29:23 +0100

[Message part 4 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hey Chris,
>
> Christopher Baines <mail <at> cbaines.net> skribis:
>
>> The Guix derivation graph isn't that deep, so I don't think this generally
>> opens lots of files, but I think it's still unnecessary to keep more files
>> than needed open.
>>
>> * guix/derivations.scm (read-derivation-from-file): Read each derivation to a
>> string, which is passed as a port to read-derivation.
>
> [...]
>
>> -      (let ((drv (call-with-input-file file read-derivation)))
>> +      (let ((drv
>> +             ;; read-derivation can call read-derivation-from-file, so to
>> +             ;; avoid having multiple open files when reading a derivation
>> +             ;; with inputs, read it in to a string first.
>> +             (call-with-input-string
>> +                 (call-with-input-file file
>> +                   get-string-all)
>> +               read-derivation)))
>
> How real is the risk of having too many open files due to a
> ‘read-derivation’ call?  (Where too many is >= 100.)
>
> You might think it’s likely because:
>
> $ guix gc -R $(guix build -d --no-grafts emacs) |grep drv$ | wc -l
> 2234
>
>
> But in fact, due to the shape of the graph + memoization (I suppose¹),
> there are at most 27 open file descriptors in this case:
>
> $ strace -o /tmp/log.strace -e openat guile -c "(use-modules (guix)) (read-derivation-from-file \"$(guix build -d --no-grafts emacs)\")"
> $ cut -d '=' -f 2- < /tmp/log.strace |sort -un | tail
>  18
>  19
>  20
>  21
>  22
>  23
>  24
>  25
>  26
>  27
>
>
> So to me the status quo is probably okay.
>
> The reason I’m paying attention to this is that allocating a string port
> plus a string for the whole contents every time would put pressure on
> memory, which is worth avoiding if we can.
>
> WDYT?

I guess there should be a way of arranging the code so that it doesn't
keep unnecessary ports, but also doesn't use strings, but that will
require some rearranging.

I think I just got thinking about this as the build coordinator was
using excessive file descriptors, but this isn't the cause.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.