GNU bug report logs -
#32980
[PATCH 0/2] Multiplexed build output from the daemon
Previous Next
Reported by: Ludovic Courtès <ludo <at> gnu.org>
Date: Sun, 7 Oct 2018 22:31:02 UTC
Severity: normal
Tags: patch
Done: ludo <at> gnu.org (Ludovic Courtès)
Bug is archived. No further changes may be made.
Full log
View this message in rfc822 format
[Message part 1 (text/plain, inline)]
Your bug report
#32980: [PATCH 0/2] Multiplexed build output from the daemon
which was filed against the guix-patches package, has been closed.
The explanation is attached below, along with your original report.
If you require more details, please reply to 32980 <at> debbugs.gnu.org.
--
32980: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=32980
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
Hello,
Ludovic Courtès <ludo <at> gnu.org> skribis:
> The downside of the protocol is that it creates quite some overhead. For
> example, when extracting a tarball, we see things like:
>
> read(13, "gmlo\0\0\0\0", 8) = 8
> read(13, "5\0\0\0\0\0\0\0", 8) = 8
> read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53
> read(13, "\0\0\0", 3) = 3
>
> That is, a 29-byte message with a 24-byte header (plus the
> 8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.)
I timed “guix build -S binutils --check”, which includes the output of
“tar xvf” followed by “tar cvf”, and the timing difference is not
noticeable (of course the peak of CPU activity is caused by xz.)
Anyway I went ahead and pushed this as
f9a8fce10f2d99efec7cb1dd0f6c5f0df9d1b2df.
I changed “@ build-output” to “@ build-log”, and also adjusted
tests/status.scm since new tests had been added to that file in the
meantime.
Ludo’.
[Message part 3 (message/rfc822, inline)]
Hello Guix!
This patch set is partly a response to the problem Ricardo raised at:
https://lists.gnu.org/archive/html/guix-devel/2018-09/msg00322.html
and partly a natural followup to the addition of (guix status):
https://issues.guix.info/issue/32837
So far the daemon would send all its output (messages it writes as well as
messages any of its build processes writes) directly as a single stream
to its clients. This had several shortcomings:
1. Clients could not distinguish messages coming from the daemon (such
as build traces or ‘guix substitute’ messages) from messages coming
from build processes. (Notably build processes could “forge” build
traces such as “@ build-started”.)
2. When max-jobs > 1, clients would receive intermingled output from all
the build processes without any way to disentangle it.
3. Build traces written by the daemon were expected to start on a new
line but a builder could write output not terminated by a newline
(e.g., “checking for …” messages from ./configure) and consequently
build traces would not start on a new line and would go unnoticed
by (guix status) and co.
With this change clients can optionally ask for “multiplexed build
output”. When it’s enabled, build output is prefixed by a special
trace, like this:
@ build-output 1234 21
checking for fcntl...
where 1234 is the PID of the build process speaking and 21 is the number
of bytes in the following build output fragment. The PID is first given
in the corresponding “@ build-started” trace.
On the client side, (guix status) is adjusted to produce events like:
(build-log #f MESSAGE)
for a message coming from the daemon, and:
(build-log PID MESSAGE)
for a message coming from PID.
The downside of the protocol is that it creates quite some overhead. For
example, when extracting a tarball, we see things like:
read(13, "gmlo\0\0\0\0", 8) = 8
read(13, "5\0\0\0\0\0\0\0", 8) = 8
read(13, "@ build-output 25935 29\ncoreutils-8.29/m4/fseterr.m4\n", 53) = 53
read(13, "\0\0\0", 3) = 3
That is, a 29-byte message with a 24-byte header (plus the
8 + 8 + 3 = 19 bytes of the underlying protocol; see ‘process-stderr’.)
Another option would be to incorporate multiplexing in the lower-level
binary protocol. However, the client side would need potentially bigger
changes: the single ‘current-build-output-port’ sink would no longer
be a good match.
Thoughts?
Ludo’.
Ludovic Courtès (2):
daemon: Support multiplexed build output.
status: Build upon multiplexed build output.
guix/scripts/build.scm | 3 +
guix/scripts/environment.scm | 1 +
guix/scripts/pack.scm | 1 +
guix/scripts/package.scm | 3 +-
guix/scripts/pull.scm | 1 +
guix/scripts/system.scm | 1 +
guix/status.scm | 165 ++++++++++++++++++++++++--------
guix/store.scm | 15 ++-
nix/libstore/build.cc | 25 +++--
nix/libstore/globals.cc | 2 +
nix/libstore/globals.hh | 9 +-
nix/libstore/worker-protocol.hh | 2 +-
nix/nix-daemon/nix-daemon.cc | 2 +-
tests/status.scm | 47 ++++++++-
tests/store.scm | 63 ++++++++++++
15 files changed, 289 insertions(+), 51 deletions(-)
--
2.19.0
This bug report was last modified 6 years and 214 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.