From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 07 12:43:30 2019 Received: (at submit) by debbugs.gnu.org; 7 Apr 2019 16:43:30 +0000 Received: from localhost ([127.0.0.1]:48647 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDAtG-00080l-GW for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:43:30 -0400 Received: from eggs.gnu.org ([209.51.188.92]:42480) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDAtE-00080X-7T for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:43:29 -0400 Received: from lists.gnu.org ([209.51.188.17]:39203) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hDAt8-00061R-TL for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:43:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56861) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDAt7-00013r-Ou for bug-guix@gnu.org; Sun, 07 Apr 2019 12:43:22 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hDAt6-00060g-Pj for bug-guix@gnu.org; Sun, 07 Apr 2019 12:43:21 -0400 Received: from world.peace.net ([64.112.178.59]:56576) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hDAt6-0005x5-JP for bug-guix@gnu.org; Sun, 07 Apr 2019 12:43:20 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDAsu-0007vo-Lh; Sun, 07 Apr 2019 12:43:08 -0400 From: Mark H Weaver To: bug-guix@gnu.org Subject: Hydra offloads often get stuck while exporting build requisites Date: Sun, 07 Apr 2019 12:41:38 -0400 Message-ID: <87mul17oo2.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 64.112.178.59 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) It has become extremely frequent for builds offloaded by hydra.gnu.org to its x86 build slave hydra.gnunet.org to get stuck indefinitely while exporting prerequisites for the build to the build slave. As I write this, both of hydra.gnunet.org's build slots (one for x86_64-linux, and one for i686-linux) are stuck in this way. Here are the stuck builds: https://hydra.gnu.org/build/3432052 https://hydra.gnu.org/build/3432472 and here are the tails of their nix build logs: --8<---------------cut here---------------start------------->8--- performing build 3432052 these derivations will be built: /gnu/store/k27i3lkb38gr3mw0mridymhik3qsg6w7-font-fira-sans-4.202.drv process 14769 acquired build slot '/var/guix/offload/hydra.gnunet.org/1' load on machine 'hydra.gnunet.org' is 1.54 (normalized: 0.385) sending 1 store item to 'hydra.gnunet.org'... exporting path `/gnu/store/gzd2cisahj50nff16p8ji813p683p5r4-font-fira-sans-4.202-checkout' --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- performing build 3432472 these derivations will be built: /gnu/store/5ivay4l7bn0sqsi7k53j4qv3kndrby17-font-google-material-design-icons-3.0.1.drv process 8985 acquired build slot '/var/guix/offload/hydra.gnunet.org/0' load on machine 'hydra.gnunet.org' is 1.98 (normalized: 0.99) sending 1 store item to 'hydra.gnunet.org'... exporting path `/gnu/store/kaj5xgnz04l4mzgj05sc5v6j7cvpbrrd-font-google-material-design-icons-3.0.1-checkout' --8<---------------cut here---------------end--------------->8--- The first of these builds has been stuck for about 10.25 hours, and the second has been stuck for 11 hours. In the recent past, I've discovered them stuck for over 24 hours. Mark From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 07 12:47:34 2019 Received: (at 35181) by debbugs.gnu.org; 7 Apr 2019 16:47:34 +0000 Received: from localhost ([127.0.0.1]:48652 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDAxC-00087Y-36 for submit@debbugs.gnu.org; Sun, 07 Apr 2019 12:47:34 -0400 Received: from world.peace.net ([64.112.178.59]:46542) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDAxA-00087J-Hy for 35181@debbugs.gnu.org; Sun, 07 Apr 2019 12:47:32 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDAx4-0007yd-OW; Sun, 07 Apr 2019 12:47:26 -0400 From: Mark H Weaver To: 35181@debbugs.gnu.org Subject: Re: Hydra offloads often get stuck while exporting build requisites Date: Sun, 07 Apr 2019 12:45:57 -0400 Message-ID: <87imvp7ogv.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) I wrote earlier: > It has become extremely frequent for builds offloaded by hydra.gnu.org > to its x86 build slave hydra.gnunet.org to get stuck indefinitely while > exporting prerequisites for the build to the build slave. > > As I write this, both of hydra.gnunet.org's build slots (one for > x86_64-linux, and one for i686-linux) are stuck in this way. Here are > the stuck builds: > > https://hydra.gnu.org/build/3432052 > https://hydra.gnu.org/build/3432472 I'll leave these builds in their stuck state for at least the next 10 hours or so, in case someone wants to investigate. Mark From debbugs-submit-bounces@debbugs.gnu.org Sun Apr 07 13:31:15 2019 Received: (at 35181) by debbugs.gnu.org; 7 Apr 2019 17:31:15 +0000 Received: from localhost ([127.0.0.1]:48662 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDBdS-0000jk-RU for submit@debbugs.gnu.org; Sun, 07 Apr 2019 13:31:15 -0400 Received: from flashner.co.il ([178.62.234.194]:51442) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDBdQ-0000jW-Oo for 35181@debbugs.gnu.org; Sun, 07 Apr 2019 13:31:13 -0400 Received: from localhost (unknown [141.226.15.124]) by flashner.co.il (Postfix) with ESMTPSA id 747A3400D1; Sun, 7 Apr 2019 17:31:06 +0000 (UTC) Date: Sun, 7 Apr 2019 20:31:05 +0300 From: Efraim Flashner To: Mark H Weaver Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites Message-ID: <20190407173105.GB1337@macbook41> References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="ghzN8eJ9Qlbqn3iT" Content-Disposition: inline In-Reply-To: <87imvp7ogv.fsf@netris.org> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --ghzN8eJ9Qlbqn3iT Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Apr 07, 2019 at 12:45:57PM -0400, Mark H Weaver wrote: > I wrote earlier: >=20 > > It has become extremely frequent for builds offloaded by hydra.gnu.org > > to its x86 build slave hydra.gnunet.org to get stuck indefinitely while > > exporting prerequisites for the build to the build slave. > >=20 > > As I write this, both of hydra.gnunet.org's build slots (one for > > x86_64-linux, and one for i686-linux) are stuck in this way. Here are > > the stuck builds: > >=20 > > https://hydra.gnu.org/build/3432052 > > https://hydra.gnu.org/build/3432472 >=20 > I'll leave these builds in their stuck state for at least the next 10 > hours or so, in case someone wants to investigate. >=20 For these two specifically it's possible that they are just really really big. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --ghzN8eJ9Qlbqn3iT Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlyqM9UACgkQQarn3Mo9 g1FT5BAAlvJADxEJbvXrF05KyRIL9hlu0YNIfejNMRcRiImtW7VkkCADQA87jJew 3e4tqNhgmEUey3eWAn+9OY3xtbHoDpLZJrr9ZPS3fnk5xi1r4vODJ/8fOHrm9EjK 1Jfl5YXed/FgN9DCoWR52jEsTvpruqOFo8GTnLVDk8+GTkL/vvwY9N5fzzaQzKu3 tAlFxQaFp/e4/2yzTx4VFGTyWXgbp3g2FvxDez4D/WvBfuFr2+VhyyETci15uxHc vAG9M2QcCslM38sajylHDJn9vlbN9mq2hw1b5sSFQ0MCBlA2zx3OELMO6IpuSAuc z04Gd6IzJdP9Pin54avVAwOcaKOpKMAikQzrzNXDDMcINHkL2DBa4Rn1n52cyncN AVGLoeZreaElSKqsb+cSOvi6YDCiYzlMCJqZJnl6f9GgiIFWbffUQNGS6MNqfCWC S6421rkrpUllHa5XxuNiwpghl6k/4FbvpjAX/RMoYxo4bfiWtySbyZH5Ri1NHD3a ub8Xc+Bn5+h/49ziQHG6PNfWKWj3o4BQFFf98oFVDHjpRRbIeFR3illQ5BUqaXaF WERrRI620mnrceXemfL1TrxBpv7II+R34xWZOoQ8AV++O0HSZUg3xlvLMshpJI5S WoULamBPcn4enIjDP9vMNpGpcbNqjGgSupUHU/9t26BuMmq3mxM= =t9Gc -----END PGP SIGNATURE----- --ghzN8eJ9Qlbqn3iT-- From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 08 02:30:22 2019 Received: (at 35181) by debbugs.gnu.org; 8 Apr 2019 06:30:22 +0000 Received: from localhost ([127.0.0.1]:48931 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDNnS-0005B4-0m for submit@debbugs.gnu.org; Mon, 08 Apr 2019 02:30:22 -0400 Received: from world.peace.net ([64.112.178.59]:47724) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDNnP-0005Ap-J3 for 35181@debbugs.gnu.org; Mon, 08 Apr 2019 02:30:20 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDNnI-0003X9-UW; Mon, 08 Apr 2019 02:30:13 -0400 From: Mark H Weaver To: Efraim Flashner Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> Date: Mon, 08 Apr 2019 02:28:41 -0400 In-Reply-To: <20190407173105.GB1337@macbook41> (Efraim Flashner's message of "Sun, 7 Apr 2019 20:31:05 +0300") Message-ID: <87ef6d6mdn.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Efraim, Efraim Flashner writes: > For these two specifically it's possible that they are just really > really big. The source checkout currently being transferred for build 3432472 (/gnu/store/=E2=80=A6-font-google-material-design-icons-3.0.1-checkout) is = 176 megabytes uncompressed, as measured by "du -s --si", which is not precisely same as NAR size, but hopefully close enough for a rough estimate. As I write this, build 3432472 been stuck here for 24 hours 15 minutes. Even if the average transfer rate were 4 kilobytes per second, it should have been done in half that time. I should also note that this is the *10th* attempt for build 3432472. I didn't realize until now, but I've manually aborted this same build 9 times before now, before filing this bug report. On its fourth attempt, it ran for just over *48* hours before I aborted it. Here's a full list of how long each of the build attempts have run: Nr Duration Machine Status -------------------------------------------------- 1 20h 23m 27s Aborted (log, raw, tail) 2 5m 28s Aborted (log, raw, tail) 3 1d 9h 54m 42s Aborted (log, raw, tail) 4 2d 0h 1m 17s Aborted (log, raw, tail) 5 8h 12m 4s Aborted (log, raw, tail) 6 1d 21h 37m 24s Aborted (log, raw, tail) 7 8h 26m 1s Aborted (log, raw, tail) 8 11h 21m 6s Aborted (log, raw, tail) 9 4h 38m 10s Aborted (log, raw, tail) 10 1d 0h 22m 19s Building (log, raw, tail) -------------------------------------------------- Source: The other build (3432052) has a very similar story. Its source checkout is slightly larger at 287 megabytes, and it's currently on its 8th attempt, which has been running for 23 hours 27 minutes as I write this. Nr Duration Machine Status -------------------------------------------------- 1 1d 4h 32m 24s Aborted (log, raw, tail) 2 2d 0h 1m 17s Aborted (log, raw, tail) 3 8h 12m 5s Aborted (log, raw, tail) 4 1d 10h 11m 48s Aborted (log, raw, tail) 5 8h 26m 11s Aborted (log, raw, tail) 6 10h 49m 44s Aborted (log, raw, tail) 7 4h 38m 20s Aborted (log, raw, tail) 8 22h 43m 9s Building (log, raw, tail) -------------------------------------------------- So far, these two builds alone have consumed a total of 15 days of Hydra's build slot time. One of the builds is on x86_64-linux and the other on i686-linux. As I recall, a similar thing happened with the mozjs-60 builds, although I didn't look closely. Mark From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 08 03:15:19 2019 Received: (at 35181) by debbugs.gnu.org; 8 Apr 2019 07:15:20 +0000 Received: from localhost ([127.0.0.1]:48939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDOUx-0006Cu-MY for submit@debbugs.gnu.org; Mon, 08 Apr 2019 03:15:19 -0400 Received: from world.peace.net ([64.112.178.59]:47770) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDOUu-0006Cd-EX for 35181@debbugs.gnu.org; Mon, 08 Apr 2019 03:15:17 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDOUo-0003qr-Jt; Mon, 08 Apr 2019 03:15:10 -0400 From: Mark H Weaver To: Efraim Flashner Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> Date: Mon, 08 Apr 2019 03:13:39 -0400 In-Reply-To: <87ef6d6mdn.fsf@netris.org> (Mark H. Weaver's message of "Mon, 08 Apr 2019 02:28:41 -0400") Message-ID: <87a7h16kap.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) The same jobs that are consistently getting stuck offloading to hydra.gnunet.org (Hydra's only functional x86 build slave at present) built successfully on armhf, with build times of 1-2 hours. With only one x86 build slave and all armhf build slaves on the same network and running the same ancient version of guix (circa 0.12.0), there are several possibilities: (1) The problem might be specific to hydra.gnunet.org or its network, or (2) it might depend on the version of guix running on the build slave, or (3) it might depend on the architecture of the build slave, or (4) something else? Also, these same jobs built without incident back in early December. Mark From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 08 04:19:31 2019 Received: (at 35181) by debbugs.gnu.org; 8 Apr 2019 08:19:31 +0000 Received: from localhost ([127.0.0.1]:48973 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDPV4-0007og-TY for submit@debbugs.gnu.org; Mon, 08 Apr 2019 04:19:31 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59332) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDPV2-0007oT-R1 for 35181@debbugs.gnu.org; Mon, 08 Apr 2019 04:19:29 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42044) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDPUw-0002T4-E4; Mon, 08 Apr 2019 04:19:23 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=49874 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hDPUv-0005ma-IJ; Mon, 08 Apr 2019 04:19:22 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 19 Germinal an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 08 Apr 2019 10:19:18 +0200 In-Reply-To: <87ef6d6mdn.fsf@netris.org> (Mark H. Weaver's message of "Mon, 08 Apr 2019 02:28:41 -0400") Message-ID: <87pnpw29kp.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org, Efraim Flashner X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Mark, Mark H Weaver skribis: > The source checkout currently being transferred for build 3432472 > (/gnu/store/=E2=80=A6-font-google-material-design-icons-3.0.1-checkout) i= s 176 > megabytes uncompressed, as measured by "du -s --si", which is not > precisely same as NAR size, but hopefully close enough for a rough > estimate. As I write this, build 3432472 been stuck here for 24 hours > 15 minutes. Even if the average transfer rate were 4 kilobytes per > second, it should have been done in half that time. This is weird, could it be that data transfers get stuck somehow? Did you try to check the status of the =E2=80=98nix-store=E2=80=99 and =E2=80= =98guix offload=E2=80=99 processes on the head node? I just checked and apparently this package builds fine on berlin. Thanks for investigating, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 08 15:42:38 2019 Received: (at 35181) by debbugs.gnu.org; 8 Apr 2019 19:42:38 +0000 Received: from localhost ([127.0.0.1]:50192 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDaA5-0003za-O3 for submit@debbugs.gnu.org; Mon, 08 Apr 2019 15:42:37 -0400 Received: from world.peace.net ([64.112.178.59]:48808) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDaA2-0003zN-T8 for 35181@debbugs.gnu.org; Mon, 08 Apr 2019 15:42:32 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDa9v-0008Fx-UR; Mon, 08 Apr 2019 15:42:24 -0400 From: Mark H Weaver To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> Date: Mon, 08 Apr 2019 15:40:51 -0400 In-Reply-To: <87pnpw29kp.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 08 Apr 2019 10:19:18 +0200") Message-ID: <87o95g5lpd.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Ludovic, Ludovic Courtès writes: > Mark H Weaver skribis: > >> The source checkout currently being transferred for build 3432472 >> (/gnu/store/…-font-google-material-design-icons-3.0.1-checkout) is 176 >> megabyte [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: gnu.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 NUMERIC_HTTP_ADDR URI: Uses a numeric IP address in URL 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Hi Ludovic, Ludovic Court=C3=A8s writes: > Mark H Weaver skribis: > >> The source checkout currently being transferred for build 3432472 >> (/gnu/store/=E2=80=A6-font-google-material-design-icons-3.0.1-checkout) = is 176 >> megabytes uncompressed, as measured by "du -s --si", which is not >> precisely same as NAR size, but hopefully close enough for a rough >> estimate. As I write this, build 3432472 been stuck here for 24 hours >> 15 minutes. Even if the average transfer rate were 4 kilobytes per >> second, it should have been done in half that time. > > This is weird, could it be that data transfers get stuck somehow? As far as I can tell, that's what seems to happen. > Did you try to check the status of the =E2=80=98nix-store=E2=80=99 and = =E2=80=98guix offload=E2=80=99 > processes on the head node? Here are the corresponding 'guix offload' processes: --8<---------------cut here---------------start------------->8--- hydra@20121227-hydra:~$ ps auxwwf | head -1; ps auxwwf | egrep -B1 'off()lo= ad' USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 8984 0.0 0.0 30784 2248 ? Ss Apr07 0:00 | \= _ /root/.guix-profile/bin/guix-daemon 8983 --max-jobs=3D1 --build-user= s-group=3Dguixbuild --disable-log-compression --gc-keep-outputs --gc-keep-d= erivations --no-substitutes --cache-failures root 8985 0.0 0.2 145532 13976 ? SLsl Apr07 0:10 | |= \_ /gnu/store/yihvhxv3xyyvl1m2cy1lnf1lyi9h76fk-guile-2.2.2/bin/guile --n= o-auto-compile /gnu/store/fkkjhida23k612naa9d4q6avqj5v3b28-guix-0.13.0-8.35= 7ab93/bin/.guix-real offload x86_64-linux 3600 1 72000 -- root 14768 0.0 0.0 30752 2356 ? Ss Apr07 0:00 | \= _ /root/.guix-profile/bin/guix-daemon 14767 --max-jobs=3D1 --build-user= s-group=3Dguixbuild --disable-log-compression --gc-keep-outputs --gc-keep-d= erivations --no-substitutes --cache-failures root 14769 0.0 0.2 145668 10912 ? SLsl Apr07 0:16 | |= \_ /gnu/store/yihvhxv3xyyvl1m2cy1lnf1lyi9h76fk-guile-2.2.2/bin/guile --n= o-auto-compile /gnu/store/fkkjhida23k612naa9d4q6avqj5v3b28-guix-0.13.0-8.35= 7ab93/bin/.guix-real offload x86_64-linux 3600 1 72000 --8<---------------cut here---------------end--------------->8--- I tried attaching to both of these offload processes with 'strace', and waited for several minutes for any system call activity. Both of them are stuck sleeping within a system call, although I don't yet know which system call: --8<---------------cut here---------------start------------->8--- root@20121227-hydra:~# strace -p 8985 Process 8985 attached - interrupt to quit restart_syscall(<... resuming interrupted call ...>^C Process 8985 detached root@20121227-hydra:~# strace -p 14769 Process 14769 attached - interrupt to quit restart_syscall(<... resuming interrupted call ...>^C Process 14769 detached --8<---------------cut here---------------end--------------->8--- Here are the 'nix-store' processes: --8<---------------cut here---------------start------------->8--- hydra@20121227-hydra:~$ ps auxwwf | head -1; ps auxwwf | egrep -A1 'hydra-(= )build' USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND hydra 8980 0.0 0.9 187332 46656 pts/5 S Apr07 0:01 | | \= _ /usr/local/bin/perl -w /usr/local/bin/hydra-build 3432472 hydra 8983 0.0 0.0 34228 464 pts/5 S Apr07 0:00 | | |= \_ nix-store --realise /gnu/store/5ivay4l7bn0sqsi7k53j4qv3kndrby17-font-= google-material-design-icons-3.0.1.drv --timeout 72000 --max-silent-time 36= 00 --option build-max-log-size 67108864 --keep-going --fallback --no-build-= output --log-type flat --print-build-trace --add-root /nix/var/nix/gcroots/= per-user/hydra/hydra-roots/y61f3cdhx31msdhkdw0kfs5pb75ycgfq-font-google-mat= erial-design-icons-3.0.1 hydra 14764 0.0 0.9 187336 46576 pts/5 S Apr07 0:01 | | \= _ /usr/local/bin/perl -w /usr/local/bin/hydra-build 3432052 hydra 14767 0.0 0.0 34228 352 pts/5 S Apr07 0:00 | | = \_ nix-store --realise /gnu/store/k27i3lkb38gr3mw0mridymhik3qsg6w7-font-= fira-sans-4.202.drv --timeout 72000 --max-silent-time 3600 --option build-m= ax-log-size 67108864 --keep-going --fallback --no-build-output --log-type f= lat --print-build-trace --add-root /nix/var/nix/gcroots/per-user/hydra/hydr= a-roots/28ncfjmplcwyzas2p3d4cy5xlzacjcnj-font-fira-sans-4.202 --8<---------------cut here---------------end--------------->8--- The 'nix-store' processes seem to be stuck sleeping in 'read', if I'm interpreting the 'strace' output correctly: --8<---------------cut here---------------start------------->8--- root@20121227-hydra:~# strace -p 8983 Process 8983 attached - interrupt to quit read(3, ^C Process 8983 detached root@20121227-hydra:~# strace -p 14767 Process 14767 attached - interrupt to quit read(3, ^C Process 14767 detached --8<---------------cut here---------------end--------------->8--- "netstat --inet --program" shows that the SSH connections are still open: --8<---------------cut here---------------start------------->8--- root@20121227-hydra:~# netstat --inet --program | grep 'hydra\.net\.in\.tum= \.' tcp 0 0 20121227-hydra.gn:53216 hydra.net.in.tum.de:ssh ESTABLI= SHED 14769/guile=20=20=20=20=20 tcp 0 0 20121227-hydra.gn:52434 hydra.net.in.tum.de:ssh ESTABLI= SHED 8985/guile=20=20=20=20=20=20 tcp 0 0 20121227-hydra.gnu.:www hydra.net.in.tum.:52104 TIME_WA= IT -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 tcp 0 0 20121227-hydra.gnu.:www hydra.net.in.tum.:52103 TIME_WA= IT -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 --8<---------------cut here---------------end--------------->8--- > I just checked and apparently this package builds fine on berlin. Also, these same jobs (e.g. same versions) have been built successfully on hydra.gnunet.org for years without difficulty. In the case of 'font-fira-sans-4.202.x86_64-linux', it has only ever been built on hydra.gnunet.org, with the last successful build on 28 September 2018. The packages haven't been updated in years, and so typically are only rebuilt during 'core-updates' cycles. They only started aborting in late March, when some other rarely-update package apparently changed to force a rebuild. However, the similar 'mozjs-60' failures happened earlier. FYI, here's the history of build attempts on Hydra: --8<---------------cut here---------------start------------->8--- hydra=3D> select case when s.machine~'^(hydra|guix)\.' then s.machine else = substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepn= r as step, case when s.busy=3D1 then 'busy' when s.status=3D0 then NULL whe= n s.status=3D1 then 'fail' when s.status=3D4 then 'abort' when s.status=3D7= then 'timeout' when s.status=3D8 then 'cfail' else '?' end as stat, regexp= _replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, da= te_trunc('day', to_timestamp(s.stoptime)) as finished from builds b, builds= teps s where b.id=3Ds.build and b.job=3D'font-fira-sans-4.202.x86_64-linux'= order by s.stoptime; machine | jobset | build | step | stat | what = | finished=20=20=20=20=20=20=20=20 ------------------+--------------+---------+------+-------+----------------= ------+------------------------ hydra.gnunet.org | master | 2362639 | 1 | | font-fira-sans-= 4.202 | 2017-11-29 00:00:00+00 hydra.gnunet.org | core-updates | 2407845 | 1 | | font-fira-sans-= 4.202 | 2018-01-02 00:00:00+00 hydra.gnunet.org | core-updates | 2674686 | 1 | | font-fira-sans-= 4.202 | 2018-05-19 00:00:00+00 hydra.gnunet.org | core-updates | 3075254 | 1 | | font-fira-sans-= 4.202 | 2018-09-28 00:00:00+00 | master | 3432052 | 1 | abort | font-fira-sans-= 4.202 | 2019-03-31 00:00:00+00 | master | 3432052 | 2 | abort | font-fira-sans-= 4.202 | 2019-04-02 00:00:00+00 | master | 3432052 | 3 | abort | font-fira-sans-= 4.202 | 2019-04-03 00:00:00+00 | master | 3432052 | 4 | abort | font-fira-sans-= 4.202 | 2019-04-05 00:00:00+00 | master | 3432052 | 5 | abort | font-fira-sans-= 4.202 | 2019-04-05 00:00:00+00 | master | 3432052 | 6 | abort | font-fira-sans-= 4.202 | 2019-04-06 00:00:00+00 | master | 3432052 | 7 | abort | font-fira-sans-= 4.202 | 2019-04-06 00:00:00+00 | master | 3432052 | 8 | busy | font-fira-sans-= 4.202 |=20 (12 rows) hydra=3D> select case when s.machine~'^(hydra|guix)\.' then s.machine else = substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepn= r as step, case when s.busy=3D1 then 'busy' when s.status=3D0 then NULL whe= n s.status=3D1 then 'fail' when s.status=3D4 then 'abort' when s.status=3D7= then 'timeout' when s.status=3D8 then 'cfail' else '?' end as stat, regexp= _replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, da= te_trunc('day', to_timestamp(s.stoptime)) as finished from builds b, builds= teps s where b.id=3Ds.build and b.job=3D'font-google-material-design-icons-= 3.0.1.i686-linux' order by s.stoptime; machine | jobset | build | step | stat | = what | finished=20=20=20=20=20=20=20=20 ------------------+--------------+---------+------+-------+----------------= --------------------------------+------------------------ chapters | master | 1834047 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2017-02-13 00:00:00+00 hydra.gnunet.org | core-updates | 1889434 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2017-03-12 00:00:00+00 | master | 2030520 | 2 | cfail | glibc-intermedi= ate-2.25 | 2017-04-30 00:00:00+00 | master | 2030520 | 1 | cfail | glibc-intermedi= ate-2.25 | 2017-04-30 00:00:00+00 | master | 2030520 | 3 | cfail | glibc-intermedi= ate-2.25 | 2017-04-30 00:00:00+00 guix.sjd.se | master | 2035120 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2017-05-04 00:00:00+00 guix.sjd.se | core-updates | 2111787 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2017-06-25 00:00:00+00 hydra.gnunet.org | master | 2128849 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2017-06-26 00:00:00+00 guix.sjd.se | core-updates | 2175161 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2017-07-20 00:00:00+00 | master | 2334641 | 1 | | font-google-mat= erial-design-icons-3.0.1.tar.gz | 2017-10-23 00:00:00+00 hydra.gnunet.org | master | 2334641 | 2 | | font-google-mat= erial-design-icons-3.0.1 | 2017-10-23 00:00:00+00 | core-updates | 2406391 | 1 | | module-import = | 2018-01-02 00:00:00+00 | core-updates | 2406391 | 2 | | module-import-c= ompiled | 2018-01-02 00:00:00+00 guix.sjd.se | core-updates | 2406391 | 3 | | font-google-mat= erial-design-icons-3.0.1 | 2018-01-02 00:00:00+00 guix.sjd.se | core-updates | 2667328 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2018-05-18 00:00:00+00 guix.sjd.se | core-updates | 3073906 | 1 | | font-google-mat= erial-design-icons-3.0.1 | 2018-09-25 00:00:00+00 | master | 3432472 | 1 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-03-29 00:00:00+00 | master | 3432472 | 2 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-03-30 00:00:00+00 | master | 3432472 | 3 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-03-31 00:00:00+00 | master | 3432472 | 4 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-04-02 00:00:00+00 | master | 3432472 | 5 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-04-03 00:00:00+00 | master | 3432472 | 6 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-04-05 00:00:00+00 | master | 3432472 | 7 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-04-05 00:00:00+00 | master | 3432472 | 8 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-04-06 00:00:00+00 | master | 3432472 | 9 | abort | font-google-mat= erial-design-icons-3.0.1 | 2019-04-06 00:00:00+00 | master | 3432472 | 10 | busy | font-google-mat= erial-design-icons-3.0.1 |=20 (26 rows) --8<---------------cut here---------------end--------------->8--- It seems that Hydra fails to record the machine name in build steps that are aborted, but I know that all of the 'aborts' above are on hydra.gnunet.org, because that's currently the only x86 build slave on Hydra. I could easily believe that this problem is specific to hydra.gnunet.org, but even if that's the case, it would be good if offloading would reliably time out before days have passed. Ideally, the timeout would be a "max-silent-time" kind of timeout, so that we don't impose an arbitrary limitation on total transfer time as long as progress is being made, and so that the timeout can be relatively short. However, even a "total-transfer-time" kind of timeout would be welcome at this point, to stop the profuse bleeding of Hydra's limited x86 build capacity. What do you think? Mark From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 08 21:07:48 2019 Received: (at 35181) by debbugs.gnu.org; 9 Apr 2019 01:07:48 +0000 Received: from localhost ([127.0.0.1]:50383 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDfEq-0001dA-0f for submit@debbugs.gnu.org; Mon, 08 Apr 2019 21:07:48 -0400 Received: from world.peace.net ([64.112.178.59]:49250) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDfEl-0001ci-7i; Mon, 08 Apr 2019 21:07:45 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDfEe-0001Nf-PJ; Mon, 08 Apr 2019 21:07:36 -0400 From: Mark H Weaver To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> Date: Mon, 08 Apr 2019 21:06:04 -0400 In-Reply-To: <87pnpw29kp.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 08 Apr 2019 10:19:18 +0200") Message-ID: <87k1g456nc.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) merge 35181 34157 thanks I looked more closely at the 'mozjs-60' failures, and I'm convinced that it's an instance of the same problem that's currently affecting these large font builds. Mozjs-60 was pushed to the master branch on 2019-01-18. It has _never_ successfully built on x86_64 or i686, although all builds were successful on armhf. See below for the complete list of build attempts of mozjs-60 on Hydra. Also of note: So far, all known instances of this problem have occurred while transferring a large directory, as opposed to a tarball. We have several packages with source tarballs _much_ larger than these problematic source checkouts, and which are updated more much frequently, and yet I've *never* seen an instance of this problem while exporting a plain file to a build slave. For example, the upstream IceCat and Firefox ESR tarballs are ~270 megabytes compressed, whereas 'font-google-material-design-icons-3.0.1' source is only ~176 megabytes _uncompressed_. I have no explanation for why the superficial form of the store item should matter here, but maybe it's a clue. I know that plain non-executable files in the store are handled somewhat differently in the Nix model than directories or executable files, the latter associated with the word "recursive", and requiring an additional layer of encoding for purposes of serialization, but I'm not sufficiently familiar with the details or relevant code. Ludovic, can you think of a reason why the file/directory distinction could be relevant to this issue? Finally: the problem seems to have been introduced into Hydra sometime between September 2018 and January 2019. September 2018 is when the last successful build of the problematic font packages was performed, and January 2019 is the first known instance of the problem. I do not currently know of any relevant data points in that time range. The last 'core-updates' merge into 'master' was on December 3rd. Mark PS: Here's the complete history of 'mozjs-60' build attempts on Hydra: First are the 'armhf' attempts, followed by i686, and x86_64. Note that the two armhf aborts happened after only 2 seconds, and surely had a different cause than this issue. --8<---------------cut here---------------start------------->8--- hydra=> select case when s.machine~'^(hydra|guix)\.' then s.machine else substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepnr as step, case when s.busy=1 then 'busy' when s.status=0 then NULL when s.status=1 then 'fail' when s.status=4 then 'abort' when s.status=7 then 'timeout' when s.status=8 then 'cfail' else '?' end as stat, regexp_replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, date_trunc('second', to_timestamp(s.stoptime)) as finished, date_trunc('second', to_timestamp(s.stoptime) - to_timestamp(s.starttime)) as duration from builds b, buildsteps s where b.id=s.build and b.job='mozjs-60.2.3-2.armhf-linux' order by s.stoptime; machine | jobset | build | step | stat | what | finished | duration --------------+--------+---------+------+-------+-------------------------+------------------------+---------- hydra-slave2 | master | 3342804 | 1 | | mozjs-60.2.3-2-checkout | 2019-01-19 12:58:52+00 | 00:23:55 hydra-slave2 | master | 3342804 | 2 | | mozjs-60.2.3-2 | 2019-01-19 15:49:37+00 | 02:50:42 | master | 3367975 | 1 | abort | mozjs-60.2.3-2 | 2019-02-13 00:03:58+00 | 00:00:02 | master | 3367975 | 2 | abort | mozjs-60.2.3-2 | 2019-02-15 15:35:45+00 | 00:00:02 hydra-slave3 | master | 3367975 | 3 | | mozjs-60.2.3-2 | 2019-02-18 16:38:08+00 | 02:46:42 (5 rows) hydra=> select case when s.machine~'^(hydra|guix)\.' then s.machine else substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepnr as step, case when s.busy=1 then 'busy' when s.status=0 then NULL when s.status=1 then 'fail' when s.status=4 then 'abort' when s.status=7 then 'timeout' when s.status=8 then 'cfail' else '?' end as stat, regexp_replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, date_trunc('second', to_timestamp(s.stoptime)) as finished, date_trunc('second', to_timestamp(s.stoptime) - to_timestamp(s.starttime)) as duration from builds b, buildsteps s where b.id=s.build and b.job='mozjs-60.2.3-2.i686-linux' order by s.stoptime; machine | jobset | build | step | stat | what | finished | duration ---------+--------+---------+------+-------+----------------+------------------------+----------------- | master | 3343511 | 1 | abort | mozjs-60.2.3-2 | 2019-01-20 20:05:16+00 | 12:11:12 | master | 3343511 | 2 | abort | mozjs-60.2.3-2 | 2019-01-23 01:52:01+00 | 2 days 05:42:55 | master | 3360985 | 1 | abort | mozjs-60.2.3-2 | 2019-02-15 19:59:42+00 | 09:31:25 | master | 3360985 | 2 | abort | mozjs-60.2.3-2 | 2019-02-16 17:37:06+00 | 05:57:15 | master | 3360985 | 3 | abort | mozjs-60.2.3-2 | 2019-02-17 17:39:49+00 | 16:06:14 | master | 3360985 | 4 | abort | mozjs-60.2.3-2 | 2019-03-03 21:50:48+00 | 00:02:19 (6 rows) hydra=> select case when s.machine~'^(hydra|guix)\.' then s.machine else substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepnr as step, case when s.busy=1 then 'busy' when s.status=0 then NULL when s.status=1 then 'fail' when s.status=4 then 'abort' when s.status=7 then 'timeout' when s.status=8 then 'cfail' else '?' end as stat, regexp_replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, date_trunc('second', to_timestamp(s.stoptime)) as finished, date_trunc('second', to_timestamp(s.stoptime) - to_timestamp(s.starttime)) as duration from builds b, buildsteps s where b.id=s.build and b.job='mozjs-60.2.3-2.x86_64-linux' order by s.stoptime; machine | jobset | build | step | stat | what | finished | duration ---------+--------+---------+------+-------+----------------+------------------------+----------------- | master | 3342528 | 1 | abort | mozjs-60.2.3-2 | 2019-01-20 20:04:50+00 | 22:25:28 | master | 3342528 | 2 | abort | mozjs-60.2.3-2 | 2019-01-23 01:51:48+00 | 2 days 05:19:35 | master | 3366691 | 1 | abort | mozjs-60.2.3-2 | 2019-02-17 17:39:59+00 | 09:21:24 (3 rows) --8<---------------cut here---------------end--------------->8--- From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 09 06:54:29 2019 Received: (at 35181) by debbugs.gnu.org; 9 Apr 2019 10:54:29 +0000 Received: from localhost ([127.0.0.1]:50598 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDoOb-0001dC-Fg for submit@debbugs.gnu.org; Tue, 09 Apr 2019 06:54:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:38613) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDoOa-0001d1-0z for 35181@debbugs.gnu.org; Tue, 09 Apr 2019 06:54:28 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37124) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDoOU-0003MO-IF; Tue, 09 Apr 2019 06:54:22 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40704 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hDoOU-0005XM-2i; Tue, 09 Apr 2019 06:54:22 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> <87o95g5lpd.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 20 Germinal an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 09 Apr 2019 12:54:20 +0200 In-Reply-To: <87o95g5lpd.fsf@netris.org> (Mark H. Weaver's message of "Mon, 08 Apr 2019 15:40:51 -0400") Message-ID: <87ftqrh2jn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Mark, Mark H Weaver skribis: > Ludovic Courtès writes: > >> Mark H Weaver skribis: >> >>> The source checkout currently being transferred for build 3432472 >>> (/gnu/store/…-font-google-material [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: gnu.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 NUMERIC_HTTP_ADDR URI: Uses a numeric IP address in URL 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Hi Mark, Mark H Weaver skribis: > Ludovic Court=C3=A8s writes: > >> Mark H Weaver skribis: >> >>> The source checkout currently being transferred for build 3432472 >>> (/gnu/store/=E2=80=A6-font-google-material-design-icons-3.0.1-checkout)= is 176 >>> megabytes uncompressed, as measured by "du -s --si", which is not >>> precisely same as NAR size, but hopefully close enough for a rough >>> estimate. As I write this, build 3432472 been stuck here for 24 hours >>> 15 minutes. Even if the average transfer rate were 4 kilobytes per >>> second, it should have been done in half that time. >> >> This is weird, could it be that data transfers get stuck somehow? > > As far as I can tell, that's what seems to happen. > >> Did you try to check the status of the =E2=80=98nix-store=E2=80=99 and = =E2=80=98guix offload=E2=80=99 >> processes on the head node? > > Here are the corresponding 'guix offload' processes: > > hydra@20121227-hydra:~$ ps auxwwf | head -1; ps auxwwf | egrep -B1 'off()= load' [...] > root 14769 0.0 0.2 145668 10912 ? SLsl Apr07 0:16 | = | \_ /gnu/store/yihvhxv3xyyvl1m2cy1lnf1lyi9h76fk-guile-2.2.2/bin/guile -= -no-auto-compile /gnu/store/fkkjhida23k612naa9d4q6avqj5v3b28-guix-0.13.0-8.= 357ab93/bin/.guix-real offload x86_64-linux 3600 1 72000 The problem is that this is an ancient Guix. In the meantime, offloading has seen relevant changes, in particular things like commit ed7b44370f71126087eb953f36aad8dc4c44109f which address stability issues with Guile-SSH (ssh dist node) that was previously used. I think we should upgrade Guix on hydra.gnu.org otherwise we=E2=80=99re lik= ely to end up chasing old bugs. > The 'nix-store' processes seem to be stuck sleeping in 'read', if I'm > interpreting the 'strace' output correctly: > > root@20121227-hydra:~# strace -p 8983 > Process 8983 attached - interrupt to quit > read(3, ^C > Process 8983 detached > root@20121227-hydra:~# strace -p 14767 > Process 14767 attached - interrupt to quit > read(3, ^C > Process 14767 detached > > > "netstat --inet --program" shows that the SSH connections are still > open: > > root@20121227-hydra:~# netstat --inet --program | grep 'hydra\.net\.in\.t= um\.' > tcp 0 0 20121227-hydra.gn:53216 hydra.net.in.tum.de:ssh ESTAB= LISHED 14769/guile=20=20=20=20=20 > tcp 0 0 20121227-hydra.gn:52434 hydra.net.in.tum.de:ssh ESTAB= LISHED 8985/guile=20=20=20=20=20=20 > tcp 0 0 20121227-hydra.gnu.:www hydra.net.in.tum.:52104 TIME_= WAIT -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 > tcp 0 0 20121227-hydra.gnu.:www hydra.net.in.tum.:52103 TIME_= WAIT -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 This could be the kind of issue that we had with (ssh dist node). It=E2=80= =99s hard to tell. > I could easily believe that this problem is specific to > hydra.gnunet.org, but even if that's the case, it would be good if > offloading would reliably time out before days have passed. That=E2=80=99s the case with commit a708de151c255712071e42e5c8284756b51768c= d, but again, the Guix installation on hydra may predate that. :-/ Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 09 06:56:36 2019 Received: (at 35181) by debbugs.gnu.org; 9 Apr 2019 10:56:36 +0000 Received: from localhost ([127.0.0.1]:50602 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDoQe-0001gb-0z for submit@debbugs.gnu.org; Tue, 09 Apr 2019 06:56:36 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39405) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDoQc-0001gP-8f for 35181@debbugs.gnu.org; Tue, 09 Apr 2019 06:56:34 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37148) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDoQW-00052e-VX; Tue, 09 Apr 2019 06:56:29 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=40706 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hDoQP-0005kv-NV; Tue, 09 Apr 2019 06:56:23 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> <87k1g456nc.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 20 Germinal an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 09 Apr 2019 12:56:20 +0200 In-Reply-To: <87k1g456nc.fsf@netris.org> (Mark H. Weaver's message of "Mon, 08 Apr 2019 21:06:04 -0400") Message-ID: <87a7gzh2gb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Mark H Weaver skribis: > Also of note: So far, all known instances of this problem have occurred > while transferring a large directory, as opposed to a tarball. > > We have several packages with source tarballs _much_ larger than these > problematic source checkouts, and which are updated more much > frequently, and yet I've *never* seen an instance of this problem while > exporting a plain file to a build slave. For example, the upstream > IceCat and Firefox ESR tarballs are ~270 megabytes compressed, whereas > 'font-google-material-design-icons-3.0.1' source is only ~176 megabytes > _uncompressed_. > > I have no explanation for why the superficial form of the store item > should matter here, but maybe it's a clue. I know that plain > non-executable files in the store are handled somewhat differently in > the Nix model than directories or executable files, the latter > associated with the word "recursive", and requiring an additional layer > of encoding for purposes of serialization, but I'm not sufficiently > familiar with the details or relevant code. > > Ludovic, can you think of a reason why the file/directory distinction > could be relevant to this issue? No, I can=E2=80=99t see why it could make a difference. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 09 14:11:22 2019 Received: (at 35181) by debbugs.gnu.org; 9 Apr 2019 18:11:22 +0000 Received: from localhost ([127.0.0.1]:51943 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDvDO-0000Ti-7f for submit@debbugs.gnu.org; Tue, 09 Apr 2019 14:11:22 -0400 Received: from world.peace.net ([64.112.178.59]:51200) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDvDL-0000TV-Kp for 35181@debbugs.gnu.org; Tue, 09 Apr 2019 14:11:21 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDvDF-0006z1-MN; Tue, 09 Apr 2019 14:11:13 -0400 From: Mark H Weaver To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> <87o95g5lpd.fsf@netris.org> <87ftqrh2jn.fsf@gnu.org> Date: Tue, 09 Apr 2019 14:09:41 -0400 In-Reply-To: <87ftqrh2jn.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 09 Apr 2019 12:54:20 +0200") Message-ID: <87pnpvrqwv.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181 Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludovic, Ludovic Court=C3=A8s writes: > The problem is that this is an ancient Guix. In the meantime, > offloading has seen relevant changes, in particular things like commit > ed7b44370f71126087eb953f36aad8dc4c44109f which address stability issues > with Guile-SSH (ssh dist node) that was previously used. > > I think we should upgrade Guix on hydra.gnu.org otherwise we=E2=80=99re l= ikely > to end up chasing old bugs. Sure, that makes sense. I also noticed the old Guix after writing my last messages, so yesterday I tried updating Hydra's Guix to 0.16.0-11, which at the time was the latest version built by Hydra. After updating, I quit and relaunched 'guix-daemon', as well as 'guix publish', hydra-queue-runner, and hydra-evaluator. With the new version of Guix, *all* offloads started failing in a strange way: it got stuck in a loop, printing endlessly repeated messages like this: process N acquired build slot '/var/guix/offload/hydra.gnunet.org/0' process N acquired build slot '/var/guix/offload/hydra.gnunet.org/0' process N acquired build slot '/var/guix/offload/hydra.gnunet.org/1' process N acquired build slot '/var/guix/offload/hydra.gnunet.org/2' process N acquired build slot '/var/guix/offload/hydra.gnunet.org/0' This is from memory because after killing the queue-runner and cancelling the 'mozjs-60' jobs (which I had intended to start building as a test), the nix output above is no longer visible on those pages, and I'm not sure offhand were to look for it. Anyway, in every offloaded build, it printed a line like the above every few seconds, with the build slot number at the end varying. I don't remember if the process number varied. This reminds that I also ran into difficulties updating 'guix' on the armhf build slaves, which are also currently stuck on an even more ancient version of Guix (circa 0.12.0). On both Hydra and its armhf build slaves, Guix is installed on top of a Debian derivative, and both 'guix' and 'guix-daemon' are launched from an environment without any Guix environment variable settings. This apparently works in ancient versions of Guix, but not recent ones. So, could the problem simply be that the 'guix' wrapper is not installing enough environment variable settings for offloading to work? Mark From debbugs-submit-bounces@debbugs.gnu.org Fri Apr 14 09:15:38 2023 Received: (at 35181-done) by debbugs.gnu.org; 14 Apr 2023 13:15:38 +0000 Received: from localhost ([127.0.0.1]:45816 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pnJH7-00009n-NG for submit@debbugs.gnu.org; Fri, 14 Apr 2023 09:15:37 -0400 Received: from mail-qt1-f170.google.com ([209.85.160.170]:38498) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pnJH5-00009V-EZ for 35181-done@debbugs.gnu.org; Fri, 14 Apr 2023 09:15:36 -0400 Received: by mail-qt1-f170.google.com with SMTP id z1so8461218qti.5 for <35181-done@debbugs.gnu.org>; Fri, 14 Apr 2023 06:15:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1681478129; x=1684070129; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=LEvYo0UXkD24W/xwJOUc82zGpMzk3d4/Lpk+MpTGjPg=; b=hQvy3u1DK16+wTcUASSQl5yT6u2BrSBnLjhUZXCq80Q1GxF7VnQH35dU9vlKAUReJI jnpqh2fYXjFeg4W6cBKpJ4TtXNo1wSKLFqHcdJkd05CLJ6M+A448yEoELPR9XwYUmCsF XlZ2DUCPQLhVg3Zs1ABlqFbOlgFfttQ+N6j30Sx87LuuC/MnVypwehPSzGCkWE/ODBpv D+efFXp8wLVo670JqdriJPE5D1kUqRFEp74FnxhscrB237+SAF7bxjH+rp3B0kCmLC7A pETnSELMgye0lI+A68Nm2BDRytkPP5UGilQwYr+yUTML9wVI1DM1lave40d+Gb8alf51 UEeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681478129; x=1684070129; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=LEvYo0UXkD24W/xwJOUc82zGpMzk3d4/Lpk+MpTGjPg=; b=ffPFFr8q25UXlPsSAevH+el06tKswdR1ZOYMHwezBU7xaWyeYIOoYgcSXabFAPRSXG 9a69UfYjjKTu+63Wea1W0R3AG5Rs96y2l4vg2uQ9yVS1O5aAdRL5RS52KI83bVUL/e9t FXo1wkQx5TGJevQ1DePayGZNEDtjsy56+tB2sguIm1+1GO9y45fKizQxWpgeaZpjMdgv 1UYKqyAvGAiczY63Pu0GZKBJrJLjkb2nHrQ33V4dMRUp4Z6/TwLeg/sUDZ1iP2WI8hmS VnNTPR3DHivgMYl0p5FEEo8nEOfwEADPaxCqXXfoyRUXG+wdM0pvBq69s/jHbJN8rim5 KWUA== X-Gm-Message-State: AAQBX9cCNMPrnlEt9OonOH6BVGJX52GwSsHgGXRNp2Gf2H+xLdnz2cdZ KM5B3aq0m3UPGe1yngZ8Rn11l+Drv7g= X-Google-Smtp-Source: AKy350bWztL6yavgDNT1k+LaXY7DmmvPa4idzCrlyzmIvf0B2LyamCi7UW4+fhg0zd8ibcLAbfxzeQ== X-Received: by 2002:ac8:5f94:0:b0:3e4:ddb1:538f with SMTP id j20-20020ac85f94000000b003e4ddb1538fmr10109705qta.18.1681478129396; Fri, 14 Apr 2023 06:15:29 -0700 (PDT) Received: from hurd (dsl-10-130-139.b2b2c.ca. [72.10.130.139]) by smtp.gmail.com with ESMTPSA id e7-20020ac84147000000b003e4e9aba4b3sm1216859qtm.73.2023.04.14.06.15.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Apr 2023 06:15:28 -0700 (PDT) From: Maxim Cournoyer To: Mark H Weaver Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> <87k1g456nc.fsf@netris.org> Date: Fri, 14 Apr 2023 09:15:27 -0400 In-Reply-To: <87k1g456nc.fsf@netris.org> (Mark H. Weaver's message of "Mon, 08 Apr 2019 21:06:04 -0400") Message-ID: <87cz46k3kg.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 35181-done Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 35181-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Heya, Mark H Weaver writes: > merge 35181 34157 > thanks I'm closing this old forgotten issue since we are no longer using hydra. Let's focus on our guix deploy/offload and cuirass-related problems :-). -- Thanks, Maxim From unknown Thu Jun 19 14:05:11 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 13 May 2023 11:24:08 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator