Subject: bug#35181: Hydra offloads often get stuck while exporting build requisites

From: Mark H Weaver <[email-masked]>
Date: Sun, 07 Apr 2019 12:41:38 -0400

It has become extremely frequent for builds offloaded by hydra.gnu.org
to its x86 build slave hydra.gnunet.org to get stuck indefinitely while
exporting prerequisites for the build to the build slave.

As I write this, both of hydra.gnunet.org's build slots (one for
x86_64-linux, and one for i686-linux) are stuck in this way.  Here are
the stuck builds:

  https://hydra.gnu.org/build/3432052
  https://hydra.gnu.org/build/3432472

and here are the tails of their nix build logs:

--8<---------------cut here---------------start------------->8---
performing build 3432052
these derivations will be built:
  /gnu/store/k27i3lkb38gr3mw0mridymhik3qsg6w7-font-fira-sans-4.202.drv
process 14769 acquired build slot '/var/guix/offload/hydra.gnunet.org/1'
load on machine 'hydra.gnunet.org' is 1.54 (normalized: 0.385)
sending 1 store item to 'hydra.gnunet.org'...
exporting path `/gnu/store/gzd2cisahj50nff16p8ji813p683p5r4-font-fira-sans-4.202-checkout'
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
performing build 3432472
these derivations will be built:
  /gnu/store/5ivay4l7bn0sqsi7k53j4qv3kndrby17-font-google-material-design-icons-3.0.1.drv
process 8985 acquired build slot '/var/guix/offload/hydra.gnunet.org/0'
load on machine 'hydra.gnunet.org' is 1.98 (normalized: 0.99)
sending 1 store item to 'hydra.gnunet.org'...
exporting path `/gnu/store/kaj5xgnz04l4mzgj05sc5v6j7cvpbrrd-font-google-material-design-icons-3.0.1-checkout'
--8<---------------cut here---------------end--------------->8---

The first of these builds has been stuck for about 10.25 hours, and the
second has been stuck for 11 hours.  In the recent past, I've discovered
them stuck for over 24 hours.

       Mark