Subject: [cuirass] Failure to write build log leads to build failure

From: Ludovic Courtès <[email-masked]>
Date: Tue, 20 Aug 2024 00:41:14 +0200

We occasionally see failed builds with truncated logs on ci.guix.  These
happens in situations where ‘cuirass remote-worker’ gets EPIPE as it
sends the build log to ‘remote-server’:

--8<---------------cut here---------------start------------->8---
2024-08-19 19:54:52 @ substituter-started /gnu/store/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv substitute
2024-08-19 19:54:52 Downloading http://141.80.167.131/nar/lzip/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv...
2024-08-19 19:54:52 
2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv                            1.8MiB/s 00:00 | 1KiB transferred
2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv                            942KiB/s 00:00 | 1KiB transferred
2024-08-19 19:54:52 
2024-08-19 19:54:52 @ substituter-succeeded /gnu/store/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv
2024-08-19 19:55:04 warning: zlib error in 'gzwrite' while sending log to 141.80.167.131: 0
2024-08-19 19:55:04 error: gdPO1dI1: unexpected error while building '/gnu/store/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv': #<&compound-exception components: (#<&external-error> #<&origin origin: "fport_write"> #<&message message: "~A"> #<&irritants irritants: ("Broken pipe")> #<&exception-with-kind-and-args kind: system-error args: ("fport_write" "~A" ("Broken pipe") (32))>)>
--8<---------------cut here---------------end--------------->8---

The “unexpected error while building” message was added in
ba3ea7947ec67e9d9585e40a30d67dbd8115e73c.

This suggests that the ‘catch #t’ around the ‘send-log’ call doesn’t
have the desired effect; instead, the ‘guard’ handler is invoked.

It looks like EPIPE comes from the pipe set up by ‘build-derivations&’
and that the underlying ‘build-derivations’ call keeps writing to even
after ‘send-log’ has closed it.

Ludo’.