From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 21 10:32:54 2019 Received: (at submit) by debbugs.gnu.org; 21 Jan 2019 15:32:54 +0000 Received: from localhost ([127.0.0.1]:40337 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glbZG-0000IX-Fn for submit@debbugs.gnu.org; Mon, 21 Jan 2019 10:32:54 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36762) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glbZD-0000II-Nu for submit@debbugs.gnu.org; Mon, 21 Jan 2019 10:32:52 -0500 Received: from lists.gnu.org ([209.51.188.17]:48333) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1glbZ6-000366-87 for submit@debbugs.gnu.org; Mon, 21 Jan 2019 10:32:45 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46958) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1glbZ5-0004Zl-94 for bug-guix@gnu.org; Mon, 21 Jan 2019 10:32:44 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1glbZ3-00032y-9K for bug-guix@gnu.org; Mon, 21 Jan 2019 10:32:43 -0500 Received: from world.peace.net ([64.112.178.59]:33438) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1glbZ3-000328-0b for bug-guix@gnu.org; Mon, 21 Jan 2019 10:32:41 -0500 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1glbZ1-0006PY-LR; Mon, 21 Jan 2019 10:32:39 -0500 From: Mark H Weaver To: bug-guix@gnu.org Subject: Hydra: mozjs-60 builds on x86_64 and i686 seemingly get stuck Date: Mon, 21 Jan 2019 10:31:46 -0500 Message-ID: <87zhruuiv6.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 64.112.178.59 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Yesterday on Hydra, I found both Intel mozjs-60 builds seemingly stuck while exporting the source checkout to hydra.gnunet.org. One had been going for ~22.5 hours, and the other for ~12 hours. I forcefully killed them and restarted them. Now I see the same thing has happened on the second attempt. Both builds have been seemingly stuck like this for about 19 hours: https://hydra.gnu.org/build/3342528 https://hydra.gnu.org/build/3343511 In both cases, the build logs are empty, and the hydra log ends with: sending 1 store item to 'hydra.gnunet.org'... exporting path `/gnu/store/j2sz7dg35vkcz38sim71jll2ix1nk554-mozjs-60.2.3-2-checkout' Of course, it's possible that they're not really stuck, but that they're merely taking a ridiculously long time to send the source checkout to the build slave. My personal checkout of the mozilla-esr60 branch, without the .hg directory, is about 2.1 gigabytes. What do you think? Mark From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 21 10:39:57 2019 Received: (at 34157) by debbugs.gnu.org; 21 Jan 2019 15:39:57 +0000 Received: from localhost ([127.0.0.1]:40345 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glbg5-0000Tu-Bp for submit@debbugs.gnu.org; Mon, 21 Jan 2019 10:39:57 -0500 Received: from flashner.co.il ([178.62.234.194]:60808) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glbg3-0000Td-6w for 34157@debbugs.gnu.org; Mon, 21 Jan 2019 10:39:55 -0500 Received: from localhost (unknown [188.120.128.87]) by flashner.co.il (Postfix) with ESMTPSA id CC5DE401FD; Mon, 21 Jan 2019 15:39:48 +0000 (UTC) Date: Mon, 21 Jan 2019 17:39:47 +0200 From: Efraim Flashner To: Mark H Weaver Subject: Re: bug#34157: Hydra: mozjs-60 builds on x86_64 and i686 seemingly get stuck Message-ID: <20190121153947.GD11658@macbook41> References: <87zhruuiv6.fsf@netris.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="AsxXAMtlQ5JHofzM" Content-Disposition: inline In-Reply-To: <87zhruuiv6.fsf@netris.org> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 User-Agent: Mutt/1.11.0 (2018-11-25) X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 34157 Cc: 34157@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --AsxXAMtlQ5JHofzM Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jan 21, 2019 at 10:31:46AM -0500, Mark H Weaver wrote: > Yesterday on Hydra, I found both Intel mozjs-60 builds seemingly stuck > while exporting the source checkout to hydra.gnunet.org. One had been > going for ~22.5 hours, and the other for ~12 hours. I forcefully killed > them and restarted them. Now I see the same thing has happened on the > second attempt. Both builds have been seemingly stuck like this for > about 19 hours: >=20 > https://hydra.gnu.org/build/3342528 > https://hydra.gnu.org/build/3343511 >=20 > In both cases, the build logs are empty, and the hydra log ends with: >=20 > sending 1 store item to 'hydra.gnunet.org'... > exporting path `/gnu/store/j2sz7dg35vkcz38sim71jll2ix1nk554-mozjs-60.2.= 3-2-checkout' >=20 > Of course, it's possible that they're not really stuck, but that they're > merely taking a ridiculously long time to send the source checkout to > the build slave. My personal checkout of the mozilla-esr60 branch, > without the .hg directory, is about 2.1 gigabytes. >=20 > What do you think? >=20 > Mark >=20 12 hours is far too long for it to tie up a build slave, sending code or not. Being silent that long doesn't trigger the auto-kill? --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --AsxXAMtlQ5JHofzM Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlxF578ACgkQQarn3Mo9 g1Fjyg/+KLxi3DQijzrjUc+eZRmbS/Sy5V9Y6Mh6li8/6xKrdsYP6WwSoQIPiuo3 Sbd+1wW98Naux/REQKM+kjlEEI2oBma5gv45A30TpicqRfaMIL+RAXDgTbdJtw5e 8i+hhlk7w5njn/4xtVjW+bOgEDm3Zj3nJEWBFtP9wSeAiF+UzjwAZ2pIXMZKmRVT Kr3z+Az7XaCcuTWkOEGe+2T4lcMe1hOF6EpYoIx28uq/2VHFMg6/zo19F7XSgy4u 2BspFAB+u96Vv10gWipzJ6+DBydstmjqZFcmNarBq5YBwLh/EKX+S2u88wB6Iiko jYZfpkP03sO5Yv4qlrBh5umvez9+o4vrDkO8OZVfS3PA2ukP3PMlrQkGiSz98pwN hMYyJdCTFTp/GECpVfMfUMcizj57YdusXtYU2W/QY4Z/4QinGbgzhoaHFGsi6+hk rCRbjEoAoX8I5l7QnrMq/nJEonQ30e/7YX01/We0St7BlTokHSsbkWwgcE+YBjBD Hbe622B+at1ryLVeDR00QNGlqcLEIlrdoAMLqoYoeUbqniYaihZpXselsDdjk3mo umr5v2y7Hqt9mxT5PuBh58S6vwRegLirvVG+58fLaggxSKRrR7/hcS+A7B3blQGc yTqcw6+Vlo56EUhf5zLUBdHSYnszy1QteL1PCRXUCKilxmO++eE= =MEwG -----END PGP SIGNATURE----- --AsxXAMtlQ5JHofzM-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 21 21:56:00 2019 Received: (at 34157) by debbugs.gnu.org; 22 Jan 2019 02:56:00 +0000 Received: from localhost ([127.0.0.1]:40686 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glmEK-00044E-6D for submit@debbugs.gnu.org; Mon, 21 Jan 2019 21:56:00 -0500 Received: from world.peace.net ([64.112.178.59]:52334) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glmEE-00043r-Pr for 34157@debbugs.gnu.org; Mon, 21 Jan 2019 21:55:55 -0500 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1glmE7-0001T7-Uw; Mon, 21 Jan 2019 21:55:48 -0500 From: Mark H Weaver To: Efraim Flashner Subject: Re: bug#34157: Hydra: mozjs-60 builds on x86_64 and i686 seemingly get stuck In-Reply-To: <20190121153947.GD11658@macbook41> (Efraim Flashner's message of "Mon, 21 Jan 2019 17:39:47 +0200") References: <87zhruuiv6.fsf@netris.org> <20190121153947.GD11658@macbook41> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) Date: Mon, 21 Jan 2019 21:54:43 -0500 Message-ID: <87sgxlv1td.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Efraim Flashner writes: > On Mon, Jan 21, 2019 at 10:31:46AM -0500, Mark H Weaver wrote: >> Yesterday on Hydra, I found both Intel mozjs-60 builds seemingly stuck >> while exporting the source checkout to hydra.gnunet.org. O [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence 0.0 NUMERIC_HTTP_ADDR URI: Uses a numeric IP address in URL X-Debbugs-Envelope-To: 34157 Cc: 34157@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Efraim Flashner writes: > On Mon, Jan 21, 2019 at 10:31:46AM -0500, Mark H Weaver wrote: >> Yesterday on Hydra, I found both Intel mozjs-60 builds seemingly stuck >> while exporting the source checkout to hydra.gnunet.org. One had been >> going for ~22.5 hours, and the other for ~12 hours. I forcefully killed >> them and restarted them. Now I see the same thing has happened on the >> second attempt. Both builds have been seemingly stuck like this for >> about 19 hours: >> >> https://hydra.gnu.org/build/3342528 >> https://hydra.gnu.org/build/3343511 >> >> In both cases, the build logs are empty, and the hydra log ends with: >> >> sending 1 store item to 'hydra.gnunet.org'... >> exporting path `/gnu/store/j2sz7dg35vkcz38sim71jll2ix1nk554-mozjs-60.2.3-2-checkout' >> >> Of course, it's possible that they're not really stuck, but that they're >> merely taking a ridiculously long time to send the source checkout to >> the build slave. My personal checkout of the mozilla-esr60 branch, >> without the .hg directory, is about 2.1 gigabytes. >> >> What do you think? >> >> Mark >> > 12 hours is far too long for it to tie up a build slave, sending code or > not. Those two builds are still occupying build slots. As I write this, they've been running for over 30 hours. I was curious whether the transfers were actually happening, even if slowly, so I looked at 'netstat' output: --8<---------------cut here---------------start------------->8--- root@20121227-hydra:~# netstat --inet --program | grep net.in.tum tcp 0 0 20121227-hydra.gn:58007 hydra.net.in.tum.de:ssh ESTABLISHED 18774/guile tcp 0 0 20121227-hydra.gn:42586 hydra.net.in.tum.de:ssh ESTABLISHED 10042/guile tcp 0 0 20121227-hydra.gn:56413 hydra.net.in.tum.de:ssh ESTABLISHED 16236/guile --8<---------------cut here---------------end--------------->8--- There are currently three builds allocated to hydra.gnunet.org (a.k.a. hydra.net.in.tum), so it appears that all three ssh connections are still active. However, even after repeating this command many times, I've never seen a non-zero "Send-Q" value. This suggests that no data is actually being sent, but that it's stuck waiting for something. I'll leave these builds alone for now, in case Ludovic wants to investigate further. > Being silent that long doesn't trigger the auto-kill? I guess that the usual timeouts do not apply to file transfers performed before the actual build takes place. Thanks, Mark From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 22 08:24:57 2019 Received: (at 34157) by debbugs.gnu.org; 22 Jan 2019 13:24:57 +0000 Received: from localhost ([127.0.0.1]:40782 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glw2z-0004v0-9p for submit@debbugs.gnu.org; Tue, 22 Jan 2019 08:24:57 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:43490) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1glw2x-0004uo-5n for 34157@debbugs.gnu.org; Tue, 22 Jan 2019 08:24:55 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 50DB24AC5; Tue, 22 Jan 2019 14:24:53 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iIUvF2wGo7io; Tue, 22 Jan 2019 14:24:52 +0100 (CET) Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 6EC633AA2; Tue, 22 Jan 2019 14:24:52 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#34157: Hydra: mozjs-60 builds on x86_64 and i686 seemingly get stuck References: <87zhruuiv6.fsf@netris.org> <20190121153947.GD11658@macbook41> <87sgxlv1td.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 3 =?utf-8?Q?Pluvi=C3=B4se?= an 227 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 22 Jan 2019 14:24:51 +0100 In-Reply-To: <87sgxlv1td.fsf@netris.org> (Mark H. Weaver's message of "Mon, 21 Jan 2019 21:54:43 -0500") Message-ID: <878szcome4.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 2.3 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Mark, Mark H Weaver skribis: > Those two builds are still occupying build slots. As I write this, > they've been running for over 30 hours. > > I was curious whether the transfers were actually happening, even if > slowly, so I l [...] Content analysis details: (2.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [185.233.100.1 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail) 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence 0.0 NUMERIC_HTTP_ADDR URI: Uses a numeric IP address in URL X-Debbugs-Envelope-To: 34157 Cc: Efraim Flashner , 34157@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Mark, Mark H Weaver skribis: > Those two builds are still occupying build slots. As I write this, > they've been running for over 30 hours. > > I was curious whether the transfers were actually happening, even if > slowly, so I l [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [185.233.100.1 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail) 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence 0.0 NUMERIC_HTTP_ADDR URI: Uses a numeric IP address in URL -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager Hi Mark, Mark H Weaver skribis: > Those two builds are still occupying build slots. As I write this, > they've been running for over 30 hours. > > I was curious whether the transfers were actually happening, even if > slowly, so I looked at 'netstat' output: > > root@20121227-hydra:~# netstat --inet --program | grep net.in.tum > tcp 0 0 20121227-hydra.gn:58007 hydra.net.in.tum.de:ssh ESTAB= LISHED 18774/guile=20=20=20=20=20 > tcp 0 0 20121227-hydra.gn:42586 hydra.net.in.tum.de:ssh ESTAB= LISHED 10042/guile=20=20=20=20=20 > tcp 0 0 20121227-hydra.gn:56413 hydra.net.in.tum.de:ssh ESTAB= LISHED 16236/guile=20=20=20=20=20 > > There are currently three builds allocated to hydra.gnunet.org > (a.k.a. hydra.net.in.tum), so it appears that all three ssh connections > are still active. However, even after repeating this command many > times, I've never seen a non-zero "Send-Q" value. This suggests that no > data is actually being sent, but that it's stuck waiting for something. Weird. > I'll leave these builds alone for now, in case Ludovic wants to > investigate further. I think you can terminate them as I=E2=80=99d rather not commit to investig= ate further now. I believe hydra.gnu.org is still running a rather old guix-daemon/offload, right? We should upgrade to the latest and greatest to make sure we=E2=80=99re after a bug that=E2=80=99s still presen= t. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 08 21:07:47 2019 Received: (at control) by debbugs.gnu.org; 9 Apr 2019 01:07:47 +0000 Received: from localhost ([127.0.0.1]:50381 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDfEo-0001d1-H9 for submit@debbugs.gnu.org; Mon, 08 Apr 2019 21:07:46 -0400 Received: from world.peace.net ([64.112.178.59]:49250) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hDfEl-0001ci-7i; Mon, 08 Apr 2019 21:07:45 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDfEe-0001Nf-PJ; Mon, 08 Apr 2019 21:07:36 -0400 From: Mark H Weaver To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#35181: Hydra offloads often get stuck while exporting build requisites References: <87mul17oo2.fsf@netris.org> <87imvp7ogv.fsf@netris.org> <20190407173105.GB1337@macbook41> <87ef6d6mdn.fsf@netris.org> <87pnpw29kp.fsf@gnu.org> Date: Mon, 08 Apr 2019 21:06:04 -0400 In-Reply-To: <87pnpw29kp.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 08 Apr 2019 10:19:18 +0200") Message-ID: <87k1g456nc.fsf@netris.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control Cc: 35181@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) merge 35181 34157 thanks I looked more closely at the 'mozjs-60' failures, and I'm convinced that it's an instance of the same problem that's currently affecting these large font builds. Mozjs-60 was pushed to the master branch on 2019-01-18. It has _never_ successfully built on x86_64 or i686, although all builds were successful on armhf. See below for the complete list of build attempts of mozjs-60 on Hydra. Also of note: So far, all known instances of this problem have occurred while transferring a large directory, as opposed to a tarball. We have several packages with source tarballs _much_ larger than these problematic source checkouts, and which are updated more much frequently, and yet I've *never* seen an instance of this problem while exporting a plain file to a build slave. For example, the upstream IceCat and Firefox ESR tarballs are ~270 megabytes compressed, whereas 'font-google-material-design-icons-3.0.1' source is only ~176 megabytes _uncompressed_. I have no explanation for why the superficial form of the store item should matter here, but maybe it's a clue. I know that plain non-executable files in the store are handled somewhat differently in the Nix model than directories or executable files, the latter associated with the word "recursive", and requiring an additional layer of encoding for purposes of serialization, but I'm not sufficiently familiar with the details or relevant code. Ludovic, can you think of a reason why the file/directory distinction could be relevant to this issue? Finally: the problem seems to have been introduced into Hydra sometime between September 2018 and January 2019. September 2018 is when the last successful build of the problematic font packages was performed, and January 2019 is the first known instance of the problem. I do not currently know of any relevant data points in that time range. The last 'core-updates' merge into 'master' was on December 3rd. Mark PS: Here's the complete history of 'mozjs-60' build attempts on Hydra: First are the 'armhf' attempts, followed by i686, and x86_64. Note that the two armhf aborts happened after only 2 seconds, and surely had a different cause than this issue. --8<---------------cut here---------------start------------->8--- hydra=> select case when s.machine~'^(hydra|guix)\.' then s.machine else substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepnr as step, case when s.busy=1 then 'busy' when s.status=0 then NULL when s.status=1 then 'fail' when s.status=4 then 'abort' when s.status=7 then 'timeout' when s.status=8 then 'cfail' else '?' end as stat, regexp_replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, date_trunc('second', to_timestamp(s.stoptime)) as finished, date_trunc('second', to_timestamp(s.stoptime) - to_timestamp(s.starttime)) as duration from builds b, buildsteps s where b.id=s.build and b.job='mozjs-60.2.3-2.armhf-linux' order by s.stoptime; machine | jobset | build | step | stat | what | finished | duration --------------+--------+---------+------+-------+-------------------------+------------------------+---------- hydra-slave2 | master | 3342804 | 1 | | mozjs-60.2.3-2-checkout | 2019-01-19 12:58:52+00 | 00:23:55 hydra-slave2 | master | 3342804 | 2 | | mozjs-60.2.3-2 | 2019-01-19 15:49:37+00 | 02:50:42 | master | 3367975 | 1 | abort | mozjs-60.2.3-2 | 2019-02-13 00:03:58+00 | 00:00:02 | master | 3367975 | 2 | abort | mozjs-60.2.3-2 | 2019-02-15 15:35:45+00 | 00:00:02 hydra-slave3 | master | 3367975 | 3 | | mozjs-60.2.3-2 | 2019-02-18 16:38:08+00 | 02:46:42 (5 rows) hydra=> select case when s.machine~'^(hydra|guix)\.' then s.machine else substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepnr as step, case when s.busy=1 then 'busy' when s.status=0 then NULL when s.status=1 then 'fail' when s.status=4 then 'abort' when s.status=7 then 'timeout' when s.status=8 then 'cfail' else '?' end as stat, regexp_replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, date_trunc('second', to_timestamp(s.stoptime)) as finished, date_trunc('second', to_timestamp(s.stoptime) - to_timestamp(s.starttime)) as duration from builds b, buildsteps s where b.id=s.build and b.job='mozjs-60.2.3-2.i686-linux' order by s.stoptime; machine | jobset | build | step | stat | what | finished | duration ---------+--------+---------+------+-------+----------------+------------------------+----------------- | master | 3343511 | 1 | abort | mozjs-60.2.3-2 | 2019-01-20 20:05:16+00 | 12:11:12 | master | 3343511 | 2 | abort | mozjs-60.2.3-2 | 2019-01-23 01:52:01+00 | 2 days 05:42:55 | master | 3360985 | 1 | abort | mozjs-60.2.3-2 | 2019-02-15 19:59:42+00 | 09:31:25 | master | 3360985 | 2 | abort | mozjs-60.2.3-2 | 2019-02-16 17:37:06+00 | 05:57:15 | master | 3360985 | 3 | abort | mozjs-60.2.3-2 | 2019-02-17 17:39:49+00 | 16:06:14 | master | 3360985 | 4 | abort | mozjs-60.2.3-2 | 2019-03-03 21:50:48+00 | 00:02:19 (6 rows) hydra=> select case when s.machine~'^(hydra|guix)\.' then s.machine else substring(s.machine from '^[^.]*') end as machine, jobset, s.build, s.stepnr as step, case when s.busy=1 then 'busy' when s.status=0 then NULL when s.status=1 then 'fail' when s.status=4 then 'abort' when s.status=7 then 'timeout' when s.status=8 then 'cfail' else '?' end as stat, regexp_replace(substr(s.drvpath,1+strpos(s.drvpath,'-')),'\.drv$','') as what, date_trunc('second', to_timestamp(s.stoptime)) as finished, date_trunc('second', to_timestamp(s.stoptime) - to_timestamp(s.starttime)) as duration from builds b, buildsteps s where b.id=s.build and b.job='mozjs-60.2.3-2.x86_64-linux' order by s.stoptime; machine | jobset | build | step | stat | what | finished | duration ---------+--------+---------+------+-------+----------------+------------------------+----------------- | master | 3342528 | 1 | abort | mozjs-60.2.3-2 | 2019-01-20 20:04:50+00 | 22:25:28 | master | 3342528 | 2 | abort | mozjs-60.2.3-2 | 2019-01-23 01:51:48+00 | 2 days 05:19:35 | master | 3366691 | 1 | abort | mozjs-60.2.3-2 | 2019-02-17 17:39:59+00 | 09:21:24 (3 rows) --8<---------------cut here---------------end--------------->8--- From unknown Tue Sep 09 16:56:43 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 13 May 2023 11:24:08 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator