From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 21 19:58:41 2019 Received: (at submit) by debbugs.gnu.org; 21 Jul 2019 23:58:41 +0000 Received: from localhost ([127.0.0.1]:59703 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpLiz-00014s-AA for submit@debbugs.gnu.org; Sun, 21 Jul 2019 19:58:41 -0400 Received: from lists.gnu.org ([209.51.188.17]:43162) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpLix-00014k-Pk for submit@debbugs.gnu.org; Sun, 21 Jul 2019 19:58:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41774) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1hpLiw-0007Yo-Jm for bug-guix@gnu.org; Sun, 21 Jul 2019 19:58:39 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hpLiv-0003g0-El for bug-guix@gnu.org; Sun, 21 Jul 2019 19:58:38 -0400 Received: from world.peace.net ([64.112.178.59]:52366) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hpLiv-0003dd-BO for bug-guix@gnu.org; Sun, 21 Jul 2019 19:58:37 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hpLiu-0001Vl-0m; Sun, 21 Jul 2019 19:58:36 -0400 From: Mark H Weaver To: bug-guix@gnu.org Subject: New linux-libre failed to build on armhf on Berlin Date: Sun, 21 Jul 2019 19:56:20 -0400 Message-ID: <87o91n3ps0.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 64.112.178.59 X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) In commit 1ad9c105c208caa9059924cbfbe4759c8101f6c9, I changed our linux-libre packages to deblob the linux-libre source tarballs ourselves, i.e. to run the deblobbing scripts provided by the linux-libre project to produce linux-libre source tarballs from the upstream linux tarballs: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=1ad9c105c208caa9059924cbfbe4759c8101f6c9 The following queries show that the updated packages built successfully on x86_64, i686, and aarch64, but they all failed on armhf: https://ci.guix.gnu.org/search?query=linux-libre-5.2.2 https://ci.guix.gnu.org/search?query=linux-libre-4.19.60 https://ci.guix.gnu.org/search?query=linux-libre-4.14.134 https://ci.guix.gnu.org/search?query=linux-libre-4.9.186 https://ci.guix.gnu.org/search?query=linux-libre-4.4.186 https://ci.guix.gnu.org/search?query=linux-libre-arm-veyron-5.2.2 https://ci.guix.gnu.org/search?query=linux-libre-arm-generic-5.2.2 https://ci.guix.gnu.org/search?query=linux-libre-arm-generic-4.19.60 https://ci.guix.gnu.org/search?query=linux-libre-arm-generic-4.14.134 https://ci.guix.gnu.org/search?query=linux-libre-arm-omap2plus-5.2.2 https://ci.guix.gnu.org/search?query=linux-libre-arm-omap2plus-4.19.60 https://ci.guix.gnu.org/search?query=linux-libre-arm-omap2plus-4.14.134 Unfortunately, I'm unable to get *any* information about what went wrong from Cuirass. None of the failed builds have associated log files, and the build details page has no useful information either. For example: https://ci.guix.gnu.org/build/1488517/details My first guess was that something went wrong in the 'computed' origin that runs the deblobbing script. However, that's apparently not the case, because all of the updated 'linux-libre-headers' packages built successfully on armhf, and those use the same source tarballs as the main 'linux-libre' packages. https://ci.guix.gnu.org/search?query=linux-libre-headers-5.2.2 https://ci.guix.gnu.org/search?query=linux-libre-headers-4.19.60 https://ci.guix.gnu.org/search?query=linux-libre-headers-4.14.134 Can someone help me find out what's going on here? Until then, I'm sorry to say that armhf-linux users will be unable to update their systems. Mark From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 21 20:38:01 2019 Received: (at control) by debbugs.gnu.org; 22 Jul 2019 00:38:01 +0000 Received: from localhost ([127.0.0.1]:59735 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpML3-00022v-6M for submit@debbugs.gnu.org; Sun, 21 Jul 2019 20:38:01 -0400 Received: from world.peace.net ([64.112.178.59]:37332) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpML1-00022k-OZ for control@debbugs.gnu.org; Sun, 21 Jul 2019 20:38:00 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hpMKw-0001my-7F; Sun, 21 Jul 2019 20:37:54 -0400 From: Mark H Weaver To: control@debbugs.gnu.org Date: Sun, 21 Jul 2019 20:35:39 -0400 Message-ID: <87imru52ix.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: severity 36754 important thanks Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) severity 36754 important thanks From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 22 12:10:55 2019 Received: (at 36754) by debbugs.gnu.org; 22 Jul 2019 16:10:55 +0000 Received: from localhost ([127.0.0.1]:33513 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpatq-0002bA-TR for submit@debbugs.gnu.org; Mon, 22 Jul 2019 12:10:55 -0400 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21298) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpatp-0002b1-0A for 36754@debbugs.gnu.org; Mon, 22 Jul 2019 12:10:54 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1563811851; cv=none; d=zoho.com; s=zohoarc; b=KfxV7kaQZTj3Go26+NG5DshdgGwmrv47Jn41doISyv0ru4DUi19gyUlSDBdj8oPLSXKnbMpH0r8afF6SQJ4U4i4AFi3ShxF///+qwD8sSlxDmcJgyKEZ7Dad48MfDsetlOhYsTAFgFHDA/E5I8ETaKGEFENYXeTmJoYFdRv9bYk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563811851; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=Xy+zDoPLaJr3BMAF8LWAlfBzADU20UzytbCj6Nto7PU=; b=GRqZ3+Y+z/b2877M527AzGrJOCqNT5ml9rmQht8RrR0IoOTTHZzKum7GoVO0m5rq76nK8Wd+YGDughyooz/6TUlOgzbtdKt6E5wAf2xMUXgPE7jJL3Hk00x39fUuBfDU+vQmV84b6GI0h5Ve6hM5CWsXvgdhx9MNmm7a35N3a78= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1563811851; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type; l=563; bh=Xy+zDoPLaJr3BMAF8LWAlfBzADU20UzytbCj6Nto7PU=; b=H53/nyRd/fubUmibObMuS8MEmZbaT54xhnLHCWMl7xFYBYs5uj0PvADAsNRsZhen agSPhWp6saNdbPdxSQelltqldceNkCuFvyyEmObHn9XUkejrCVCbElw/XFCscwyLjlY 1Auv1SaezcQH8fIpNM0DhuVox29IY63XDtGgoeR4= Received: from localhost (141.80.247.250 [141.80.247.250]) by mx.zohomail.com with SMTPS id 1563811849061428.6631695868524; Mon, 22 Jul 2019 09:10:49 -0700 (PDT) References: <87o91n3ps0.fsf@netris.org> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: mhw@netris.org Subject: Re: bug#36754: New linux-libre failed to build on armhf on Berlin In-reply-to: <87o91n3ps0.fsf@netris.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 22 Jul 2019 18:10:46 +0200 Message-ID: <87pnm2cant.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Mark H Weaver writes: > Unfortunately, I'm unable to get *any* information about what went wrong > from Cuirass. None of the failed builds have associated log files, and > the build details page has no useful information either. For example: > > https://ci.guix.gnu.org/build/1488517/details On that page I see a link to the build log, but it appears to be truncated: https://ci.guix.gnu.org/log/33hv7mij9bqqgf5hqwrw14106z9zgav9-linux-libre-5.2.2 Maybe the build node died before the build could be completed? -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 22 13:14:07 2019 Received: (at 36754) by debbugs.gnu.org; 22 Jul 2019 17:14:07 +0000 Received: from localhost ([127.0.0.1]:33567 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpbt0-0004Ab-Ng for submit@debbugs.gnu.org; Mon, 22 Jul 2019 13:14:07 -0400 Received: from world.peace.net ([64.112.178.59]:40750) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpbsz-00049t-9S for 36754@debbugs.gnu.org; Mon, 22 Jul 2019 13:14:05 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hpbss-0007oE-Pw; Mon, 22 Jul 2019 13:13:58 -0400 From: Mark H Weaver To: Ricardo Wurmus Subject: Re: bug#36754: New linux-libre failed to build on armhf on Berlin In-Reply-To: <87o91n3ps0.fsf@netris.org> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> Date: Mon, 22 Jul 2019 13:13:11 -0400 Message-ID: <87zhl656xk.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ricardo, Interesting. I distinctly remember that there was no log file when I looked last time. Hmm. Anyway, it seems that now, all of the failed builds have either build logs available or else information about which dependency failed. I don't remember seeing any of this last time, but I'm glad to see it now. A pattern has now emerged, but I don't know what it means. All of the armhf kernel builds failed except for linux-libre-arm-veyron-5.2.2, which succeeded: https://ci.guix.gnu.org/build/1488502/details (arm-veyron-5.2.2) Apart from this anomalous success, all of the armhf 5.2.2 and 4.19.60 have a truncated log file: https://ci.guix.gnu.org/build/1488517/details (5.2.2) https://ci.guix.gnu.org/build/1488503/details (4.19.60) https://ci.guix.gnu.org/build/1488513/details (arm-generic-5.2.2) https://ci.guix.gnu.org/build/1488519/details (arm-generic-4.19.60) https://ci.guix.gnu.org/build/1488504/details (arm-omap2plus-5.2.2) https://ci.guix.gnu.org/build/1488501/details (arm-omap2plus-4.19.60) This pattern seems too regular to be a coincidence. Can we find out which build machines were used for these builds? All of the 4.14.134 builds failed in the deblobbing step, due to timeout (1 hour of silence) while packing the linux-libre tarball: https://ci.guix.gnu.org/build/1488514/details (4.14.134) https://ci.guix.gnu.org/build/1488515/details (arm-generic-4.14.134) https://ci.guix.gnu.org/build/1488512/details (arm-omap2plus-4.14.134) I'm not sure how to deal with this. This is a computed origin, not a normal package, and so I don't see a way to configure a longer timeout. Perhaps I should make the tarball packing and unpacking operations verbose, to work around the issue. Of course that's our usual practice, but I find it suboptimal because any warnings will be buried in a mountain of uninteresting output. Thoughts? Anyway, thanks for looking into it. Mark From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 23 12:47:00 2019 Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 16:47:00 +0000 Received: from localhost ([127.0.0.1]:35547 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpxwK-0006HI-0L for submit@debbugs.gnu.org; Tue, 23 Jul 2019 12:47:00 -0400 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:38271) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpxwG-0006H2-0w for 36754@debbugs.gnu.org; Tue, 23 Jul 2019 12:46:56 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 99FE420D2B; Tue, 23 Jul 2019 12:46:49 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Tue, 23 Jul 2019 12:46:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h= from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-type:content-transfer-encoding; s=fm3; bh= 94RpzwMc+v0Cde5lZiZ6F4Nzf2pdlrmrVWsKdiyo2Sw=; b=G3P67n9UZQKYNgD4 7slGqZHnF1l/mu6HvvuR/toEfqgQGvayEZK7YYfUydbTW41QnQRe1KfAM7ZK0JY8 W8+B97Qovxc6b7Dnm5WKloqB73Wd8gPkTT87gm+ZeSdaiD/67XT0FKI2xHULsYUM +ix88ioucbI7cqs7+JAWHk56Z4ENeWcfrMIA4JlEuxt5Qch24+cpRuBST8afZ3VW hzE7SnPUr6dVoCBMMXSkTJqEyHqhxriGKyFL6I5tODQ0F8/M5rtV5TkmDgb+BDL/ JtieTHQ5DOnzxKTgGcMakSJyTRLQrOnXWbuxIM5eOEPQGkCEU9Yy9o20oo4bQPW2 VdEOSQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=94RpzwMc+v0Cde5lZiZ6F4Nzf2pdlrmrVWsKdiyo2 Sw=; b=ndbjgeQBR4fdqEDDFDiHWics2ptAgX4Wq+sprtfdOSVUs0IXHoHDj5ipG g8IhTJPovAqFQhlZU5no37ghZwi8b0MXRDx+U7gtnKe6f2n1p2yFSET8l1GiOXL1 JeekS7dZ0MfRklYPKPFrWN56/dExxfD+0rclGWIIWJWhcJgc5A7zyVB7yR1/3kZt DNVJOnmJvtCwFBUMEl0sSQ++FWZRz7vr6s8b6YUbDZh96RctiPJI2LoH90d6iGxp PFERWzaz/3O4niLNN4U1FX0RpA9uwVsfhGoNuchu9g3DiFJV9I01M4zdx8jUZNPi 39xC//BJ/giCs+4wlTKkWrgSXO3Gg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrjeekgddutdegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefhvffujghffgffkfggtgfgsehtqh ertddtreejnecuhfhrohhmpeforghrihhushcuuegrkhhkvgcuoehmsggrkhhkvgesfhgr shhtmhgrihhlrdgtohhmqeenucffohhmrghinhepghhnuhdrohhrghenucfkphepiedvrd duiedrvddviedrudegtdenucfrrghrrghmpehmrghilhhfrhhomhepmhgsrghkkhgvsehf rghsthhmrghilhdrtghomhenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from localhost (140.226.16.62.customer.cdi.no [62.16.226.140]) by mail.messagingengine.com (Postfix) with ESMTPA id 8AB8080059; Tue, 23 Jul 2019 12:46:48 -0400 (EDT) From: Marius Bakke To: Mark H Weaver , Ricardo Wurmus Subject: Re: bug#36754: New linux-libre failed to build on armhf on Berlin In-Reply-To: <87zhl656xk.fsf@netris.org> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> User-Agent: Notmuch/0.29.1 (https://notmuchmail.org) Emacs/26.2 (x86_64-pc-linux-gnu) Date: Tue, 23 Jul 2019 18:46:46 +0200 Message-ID: <87pnm04s21.fsf@devup.no> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36754 Cc: 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Mark H Weaver writes: > Hi Ricardo, > > Interesting. I distinctly remember that there was no log file when I > looked last time. Hmm. > > Anyway, it seems that now, all of the failed builds have either build > logs available or else information about which dependency failed. I > don't remember seeing any of this last time, but I'm glad to see it now. > > A pattern has now emerged, but I don't know what it means. All of the > armhf kernel builds failed except for linux-libre-arm-veyron-5.2.2, > which succeeded: > > https://ci.guix.gnu.org/build/1488502/details (arm-veyron-5.2.2) > > Apart from this anomalous success, all of the armhf 5.2.2 and 4.19.60 > have a truncated log file: > > https://ci.guix.gnu.org/build/1488517/details (5.2.2) > https://ci.guix.gnu.org/build/1488503/details (4.19.60) > https://ci.guix.gnu.org/build/1488513/details (arm-generic-5.2.2) > https://ci.guix.gnu.org/build/1488519/details (arm-generic-4.19.60) > https://ci.guix.gnu.org/build/1488504/details (arm-omap2plus-5.2.2) > https://ci.guix.gnu.org/build/1488501/details (arm-omap2plus-4.19.60) > > This pattern seems too regular to be a coincidence. Can we find out > which build machines were used for these builds? I tried building 5.2.2 'interactively' on Berlin, and got an SSH error: CC [M] net/openvswitch/vport-geneve.o CC [M] net/openvswitch/vport-gre.o LD [M] net/openvswitch/openvswitch.o ;;; [2019/07/23 05:14:53.501502, 0] read_from_channel_port: [GSSH ERROR] Er= ror reading from the channel: # Backtrace: 16 (apply-smob/1 #) In ice-9/boot-9.scm: 705:2 15 (call-with-prompt _ _ #) In ice-9/eval.scm: 619:8 14 (_ #(#(#))) In guix/ui.scm: 1747:12 13 (run-guix-command _ . _) In guix/scripts/offload.scm: 781:22 12 (guix-offload . _) In ice-9/boot-9.scm: 829:9 11 (catch _ _ # =E2=80=A6) 829:9 10 (catch _ _ # =E2=80=A6) In guix/scripts/offload.scm: 580:19 9 (process-request _ _ _ _ #:print-build-trace? _ # _ # _) 531:6 8 (call-with-timeout _ _ _) 361:2 7 (transfer-and-offload # =E2=80=A6) In ice-9/boot-9.scm: 829:9 6 (catch _ _ # =E2=80=A6) In guix/scripts/offload.scm: 385:6 5 (_) In guix/store.scm: 1203:15 4 (_ # _ _) 692:11 3 (process-stderr # _) In guix/serialization.scm: 87:11 2 (read-int _) 73:12 1 (get-bytevector-n* # =E2=80=A6) In unknown file: 0 (get-bytevector-n # =E2=80=A6) ERROR: In procedure get-bytevector-n: Throw to key `guile-ssh-error' with args `("read_from_channel_port" "Error = reading from the channel" # #f)'. guix build: error: build of `/gnu/store/yfns7ga468vmv9jn72snk79b16p8mhfa-li= nux-libre-5.2.2.drv' failed real 637m24.906s user 0m6.661s sys 0m0.897s Unfortunately I failed to record which machine was used and don't know a way to find out after the fact. From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 23 13:35:36 2019 Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 17:35:36 +0000 Received: from localhost ([127.0.0.1]:35579 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpyhL-0001Fq-Sk for submit@debbugs.gnu.org; Tue, 23 Jul 2019 13:35:36 -0400 Received: from world.peace.net ([64.112.178.59]:48912) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpyhK-0001FV-Hn; Tue, 23 Jul 2019 13:35:34 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hpyhE-00080a-PP; Tue, 23 Jul 2019 13:35:28 -0400 From: Mark H Weaver To: Marius Bakke Subject: Re: bug#36754: SSH connections to hydra-slave{1,2,3} fail during builds (was: New linux-libre failed to build on armhf on Berlin) In-Reply-To: <87pnm04s21.fsf@devup.no> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> Date: Tue, 23 Jul 2019 13:33:09 -0400 Message-ID: <87ef2gr6z3.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , 36754@debbugs.gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) retitle 36754 SSH connections to hydra-slave{1,2,3} fail during builds thanks Hi, I've added Ludovic to the CC list, since he recently added hydra-slave{1,2,3} to Berlin. Marius wrote: > I tried building 5.2.2 'interactively' on Berlin, and got an SSH error: >=20 > CC [M] net/openvswitch/vport-geneve.o > CC [M] net/openvswitch/vport-gre.o > LD [M] net/openvswitch/openvswitch.o > ;;; [2019/07/23 05:14:53.501502, 0] read_from_channel_port: [GSSH ERROR] = Error reading from the channel: # > Backtrace: > 16 (apply-smob/1 #) > In ice-9/boot-9.scm: > 705:2 15 (call-with-prompt _ _ #) > In ice-9/eval.scm: > 619:8 14 (_ #(#(#))) > In guix/ui.scm: > 1747:12 13 (run-guix-command _ . _) > In guix/scripts/offload.scm: > 781:22 12 (guix-offload . _) > In ice-9/boot-9.scm: > 829:9 11 (catch _ _ # =E2=80=A6) > 829:9 10 (catch _ _ # =E2=80=A6) > In guix/scripts/offload.scm: > 580:19 9 (process-request _ _ _ _ #:print-build-trace? _ # _ # _) > 531:6 8 (call-with-timeout _ _ _) > 361:2 7 (transfer-and-offload # =E2=80=A6) > In ice-9/boot-9.scm: > 829:9 6 (catch _ _ # =E2=80=A6) > In guix/scripts/offload.scm: > 385:6 5 (_) > In guix/store.scm: > 1203:15 4 (_ # _ _) > 692:11 3 (process-stderr # _) > In guix/serialization.scm: > 87:11 2 (read-int _) > 73:12 1 (get-bytevector-n* # =E2=80=A6) > In unknown file: > 0 (get-bytevector-n # =E2=80=A6) >=20 > ERROR: In procedure get-bytevector-n: > Throw to key `guile-ssh-error' with args `("read_from_channel_port" "Erro= r reading from the channel" # #f)'. > guix build: error: build of `/gnu/store/yfns7ga468vmv9jn72snk79b16p8mhfa-= linux-libre-5.2.2.drv' failed >=20 > real 637m24.906s > user 0m6.661s > sys 0m0.897s Thank you, this is helpful. > Unfortunately I failed to record which machine was used and don't know a > way to find out after the fact. I believe it was hydra-slave2, one of the three armhf machines that I host which were formerly part of hydra.gnu.org's build farm and were recently added to Berlin by Ludovic. I checked hydra-slave{1,2,3} for build log files corresponding to the derivation above, and found that all three of them have been attempted recently: hydra-slave2 attempted to build it on July 23 08:07 UTC. hydra-slave3 attempted to build it on July 22 16:40 UTC. hydra-slave1 attempted to build it on July 22 04:44 UTC. To be precise, each of those dates correspond to the end of the build attempt. All three build logs are truncated on the build machine as well, with no error message at the end. I now believe that these failures are related to the newly added armhf build slaves, and that they have nothing to do with the recent changes to our linux-libre packages. Well, except for the silence timeout that sometimes happens on slower machines while deblobbing linux-libre. That's a separate issue. Thanks, Mark From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 23 13:51:49 2019 Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 17:51:49 +0000 Received: from localhost ([127.0.0.1]:35595 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpyx3-0003nd-Dw for submit@debbugs.gnu.org; Tue, 23 Jul 2019 13:51:49 -0400 Received: from world.peace.net ([64.112.178.59]:49008) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hpyx2-0003nR-1N for 36754@debbugs.gnu.org; Tue, 23 Jul 2019 13:51:48 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hpywv-0008A6-Gs; Tue, 23 Jul 2019 13:51:41 -0400 From: Mark H Weaver To: Marius Bakke Subject: Re: bug#36754: SSH connections to hydra-slave{1,2,3} fail during builds (was: New linux-libre failed to build on armhf on Berlin) In-Reply-To: <87ef2gr6z3.fsf@netris.org> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> Date: Tue, 23 Jul 2019 13:49:24 -0400 Message-ID: <878ssor680.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , 36754@debbugs.gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) I wrote earlier: > I now believe that these failures are related to the newly added armhf > build slaves, and that they have nothing to do with the recent changes > to our linux-libre packages. I should mention that the armhf build slaves are on a private network, and I use my public-facing internet server to forward TCP connections to them, using the following entries in /etc/inetd.conf: --8<---------------cut here---------------start------------->8--- # TCP-level forwards for SSH connections to build machines for the GNU # Guix build farm: 7275 stream tcp nowait nobody /bin/nc /bin/nc -w 10 172.19.189.11 7275 7276 stream tcp nowait nobody /bin/nc /bin/nc -w 10 172.19.189.12 7276 7274 stream tcp nowait nobody /bin/nc /bin/nc -w 10 172.19.189.13 7274 --8<---------------cut here---------------end--------------->8--- It's possible that this arrangement is somehow part of the problem. However, note that nothing has changed here in several years, and it worked fine on hydra.gnu.org. The build slaves were running a *very* old version of Guix though. It seems likely that the new Guile-SSH code doesn't cope well with this setup. Mark From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 23 17:26:41 2019 Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 21:26:41 +0000 Received: from localhost ([127.0.0.1]:35709 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hq2Iy-00084C-Nb for submit@debbugs.gnu.org; Tue, 23 Jul 2019 17:26:41 -0400 Received: from eggs.gnu.org ([209.51.188.92]:46632) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hq2Iv-00083x-Ac for 36754@debbugs.gnu.org; Tue, 23 Jul 2019 17:26:38 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36806) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hq2Ip-0003UL-LX; Tue, 23 Jul 2019 17:26:31 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47812 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hq2Io-00073T-Hh; Tue, 23 Jul 2019 17:26:31 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 5 Thermidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 23 Jul 2019 23:26:27 +0200 In-Reply-To: <878ssor680.fsf@netris.org> (Mark H. Weaver's message of "Tue, 23 Jul 2019 13:49:24 -0400") Message-ID: <87lfwo1lz0.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , Marius Bakke , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Mark, Mark H Weaver skribis: > I wrote earlier: >> I now believe that these failures are related to the newly added armhf >> build slaves, and that they have nothing to do with the recent changes >> to our linux-libre packages. > > I should mention that the armhf build slaves are on a private network, > and I use my public-facing internet server to forward TCP connections to > them, using the following entries in /etc/inetd.conf: > > # TCP-level forwards for SSH connections to build machines for the GNU > # Guix build farm: > 7275 stream tcp nowait nobody /bin/nc /bin/nc -w 10 172.19.189.= 11 7275 > 7276 stream tcp nowait nobody /bin/nc /bin/nc -w 10 172.19.189.= 12 7276 > 7274 stream tcp nowait nobody /bin/nc /bin/nc -w 10 172.19.189.= 13 7274 > > It's possible that this arrangement is somehow part of the problem. > However, note that nothing has changed here in several years, and it > worked fine on hydra.gnu.org. The build slaves were running a *very* > old version of Guix though. It seems likely that the new Guile-SSH code > doesn't cope well with this setup. I noticed that connections to the machines were unstable (using OpenSSH=E2=80=99s client). That is, the connection would eventually =E2=80= =9Chang=E2=80=9D, apparently several times a day. Currently we have an SSH tunnel set up on berlin to connect to each of these machines via overdrive1.guixsd.org. This setup proved to be robust in the past (we used it to connect to another build machine), so I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the ne= twork. It=E2=80=99s hard to tell exactly what, though. Ideas? If it=E2=80=99s causing build failures, I=E2=80=99m afraid we=E2=80=99ll ha= ve to comment out those machines from berlin=E2=80=99s machines.scm until we=E2=80=99ve figur= ed it out. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 23 17:55:25 2019 Received: (at 36754) by debbugs.gnu.org; 23 Jul 2019 21:55:26 +0000 Received: from localhost ([127.0.0.1]:35738 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hq2kn-0000Rs-Ef for submit@debbugs.gnu.org; Tue, 23 Jul 2019 17:55:25 -0400 Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21305) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hq2kl-0000Rh-GK for 36754@debbugs.gnu.org; Tue, 23 Jul 2019 17:55:24 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1563918917; cv=none; d=zoho.com; s=zohoarc; b=ZQcKZa7V5lwm+FsJh7VLUlJwCPkd+xP8HVW2NNifBc5we9AF6BES95eFA2ZvhcEruxWJHXmUAq3TqyT3jX2mi4ABdW7g2RX7fPxUeiM1gCDVa07Ch4LHAa9e8qZ8dpWVGpMq5CyEftX8M5lll2P4THUFc+m3fTBKVOO0c6rtS4A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563918917; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=7eGyOrb+srBmeDwnQGRsxttF5NjD+kOyTQHbOH+8vXw=; b=Qk38l7WFDJ2DZ4K/XeL0RamnZen5aoBmtmHKP66KFEqBqppOrx3RgA/q1u+uSS0crxreFXYLxjw79Sto+2oWgXOzx1r4SO36TugbjBo4Zjr9h3nU6WPnO8ZK5CyG9gyKWNxKAf8niAK0l54ddEGEfjMoelPs6LoxjFtVKVriAtU= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1563918917; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; l=539; bh=7eGyOrb+srBmeDwnQGRsxttF5NjD+kOyTQHbOH+8vXw=; b=J7y0/0At+cm83mCodU+viysyRjSZO+FHWssAom/8WcgPBjYZzaYqH0COSvrrb+WN BzWfwe5KNAHo++TbtaPYRSO/EbfHQbF6kd3m1Ed2YncHpRsyeltfODQATedQsFmNQA2 msV+yk+ls+1tMln/ciJto7LrN5m2zD8F/iTdDxbI= Received: from localhost (p54AD4F38.dip0.t-ipconnect.de [84.173.79.56]) by mx.zohomail.com with SMTPS id 1563918916979714.2313585498698; Tue, 23 Jul 2019 14:55:16 -0700 (PDT) References: <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-reply-to: <87lfwo1lz0.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 23 Jul 2019 23:55:13 +0200 Message-ID: <87pnm0bem6.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Mark H Weaver , Marius Bakke , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Currently we have an SSH tunnel set up on berlin to connect to each of > these machines via overdrive1.guixsd.org. This setup proved to be > robust in the past (we used it to connect to another build machine), so > I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the = network. It=E2=80=99s hard to > tell exactly what, though. FWIW by the end of this week we should have the firewall changes implemented so we can do without the SSH tunnel. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 24 07:12:05 2019 Received: (at 36754) by debbugs.gnu.org; 24 Jul 2019 11:12:05 +0000 Received: from localhost ([127.0.0.1]:36018 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hqFBl-0008CI-5k for submit@debbugs.gnu.org; Wed, 24 Jul 2019 07:12:05 -0400 Received: from world.peace.net ([64.112.178.59]:51398) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hqFBi-0008Bm-OV for 36754@debbugs.gnu.org; Wed, 24 Jul 2019 07:12:04 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hqFBc-0006Ii-Dt; Wed, 24 Jul 2019 07:11:56 -0400 From: Mark H Weaver To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-Reply-To: <87lfwo1lz0.fsf@gnu.org> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> Date: Wed, 24 Jul 2019 07:09:38 -0400 Message-ID: <87ef2fpu2a.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , Marius Bakke , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludovic, Ludovic Court=C3=A8s wrote: > I noticed that connections to the machines were unstable (using > OpenSSH=E2=80=99s client). That is, the connection would eventually =E2= =80=9Chang=E2=80=9D, > apparently several times a day. > > Currently we have an SSH tunnel set up on berlin to connect to each of > these machines via overdrive1.guixsd.org. This setup proved to be > robust in the past (we used it to connect to another build machine), so > I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the = network. It=E2=80=99s hard to > tell exactly what, though. > > Ideas? Okay, I'll look into it. I'm very busy with something else for the next couple of days, but I'll try to get to it in the next week. > If it=E2=80=99s causing build failures, I=E2=80=99m afraid we=E2=80=99ll = have to comment out > those machines from berlin=E2=80=99s machines.scm until we=E2=80=99ve fig= ured it out. Agreed. Thanks, Mark From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 24 10:56:46 2019 Received: (at 36754) by debbugs.gnu.org; 24 Jul 2019 14:56:46 +0000 Received: from localhost ([127.0.0.1]:37856 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hqIhC-0007vK-4F for submit@debbugs.gnu.org; Wed, 24 Jul 2019 10:56:46 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53739) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hqIh9-0007v7-Rk for 36754@debbugs.gnu.org; Wed, 24 Jul 2019 10:56:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52253) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hqIh4-0002R7-8D; Wed, 24 Jul 2019 10:56:38 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49850 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hqIh3-0006ug-Nf; Wed, 24 Jul 2019 10:56:38 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Mark H Weaver Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 Thermidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 24 Jul 2019 16:56:35 +0200 In-Reply-To: <87ef2fpu2a.fsf@netris.org> (Mark H. Weaver's message of "Wed, 24 Jul 2019 07:09:38 -0400") Message-ID: <87lfwnxyzg.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , Marius Bakke , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, Mark H Weaver skribis: > Ludovic Court=C3=A8s wrote: >> I noticed that connections to the machines were unstable (using >> OpenSSH=E2=80=99s client). That is, the connection would eventually =E2= =80=9Chang=E2=80=9D, >> apparently several times a day. >> >> Currently we have an SSH tunnel set up on berlin to connect to each of >> these machines via overdrive1.guixsd.org. This setup proved to be >> robust in the past (we used it to connect to another build machine), so >> I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the= network. It=E2=80=99s hard to >> tell exactly what, though. >> >> Ideas? > > Okay, I'll look into it. I'm very busy with something else for the next > couple of days, but I'll try to get to it in the next week. OK! >> If it=E2=80=99s causing build failures, I=E2=80=99m afraid we=E2=80=99ll= have to comment out >> those machines from berlin=E2=80=99s machines.scm until we=E2=80=99ve fi= gured it out. > > Agreed. I=E2=80=99ve commented them out now. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 01 10:09:15 2019 Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 14:09:15 +0000 Received: from localhost ([127.0.0.1]:54816 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htBla-0004H9-Im for submit@debbugs.gnu.org; Thu, 01 Aug 2019 10:09:15 -0400 Received: from wout2-smtp.messagingengine.com ([64.147.123.25]:44137) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htBlW-0004Gs-Di for 36754@debbugs.gnu.org; Thu, 01 Aug 2019 10:09:13 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 8BDF02E0; Thu, 1 Aug 2019 10:09:04 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 01 Aug 2019 10:09:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h= from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-type; s=fm3; bh=FlvabjkkI+fID3UZ2mJy/KZEJ8 Fus3U3YTjoyxuZsFM=; b=KJmAJPJ4IrM+6LY1Nrny1EdH1nAaApWMbLsFfFfS5v F/BBzgwYHo+7lRRYLv4EKLeLJZDXk+g3Bk1BxQvzWXPUX4DEFpOE1p2SyDE8R9Rf qrGwkWhNPcgGRC7HZrbSUk/4fwrRMpt69FE/6qvlNrvNmFaahIRICWm13SH102L0 IENNUt+rhvWxy7qOk33vafaR0lBb0m3Tj/Cbf3yVjuu0Q4CNa0+437Q+OaFeNsGa ltMA4gZwqhnGxeVQmn6OOBM6zz3IkvYnmEOBbpY4qLcjP3IiAeJShy5NTjG01L9y +eeKUj78k1b3o/n2b7xU46mnKrFYj1xwOKfwMg3X+avw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=Flvabj kkI+fID3UZ2mJy/KZEJ8Fus3U3YTjoyxuZsFM=; b=e/E9vD+6uKt5a0dDbKlCB5 Tw32Gi1argKYHsG2CEI1MONwSFGbko2yjNZLa//U9UrfDrJRjP9GKOnsBZkaMgCE F+zeOUgl7VozYkb+ucd1OruBf09AHGYARAdkZC3U94QNRHlVJby3q5HNPNzIxpy6 SpxNoc4uSNXWGb3iGDU+wHenZONPH8Z1+e30CrjiNOpu9sOOcn+lODU9hizweS3n hQ3rYPTbjfhCgyiW1RsWU1VZHUWlJF4dM3wC4MrGQp/F0vtmRc6FQXWqDLlK699L ksRFSs5Pu8QPqRxbUfs/S5kxB4YCk8jhJu7iMcM8pOpTlAXqdaHa5Begfejs/xfg == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrleejgdejudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffujghffgffkfggtgesghdtreertdertdenucfhrhhomhepofgrrhhiuhhs uceurghkkhgvuceomhgsrghkkhgvsehfrghsthhmrghilhdrtghomheqnecuffhomhgrih hnpehgnhhurdhorhhgnecukfhppeeivddrudeirddvvdeirddugedtnecurfgrrhgrmhep mhgrihhlfhhrohhmpehmsggrkhhkvgesfhgrshhtmhgrihhlrdgtohhmnecuvehluhhsth gvrhfuihiivgeptd X-ME-Proxy: Received: from localhost (140.226.16.62.customer.cdi.no [62.16.226.140]) by mail.messagingengine.com (Postfix) with ESMTPA id 20C8780064; Thu, 1 Aug 2019 10:09:03 -0400 (EDT) From: Marius Bakke To: Ludovic =?utf-8?Q?Court=C3=A8s?= , Mark H Weaver Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-Reply-To: <87lfwnxyzg.fsf@gnu.org> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> <87lfwnxyzg.fsf@gnu.org> User-Agent: Notmuch/0.29.1 (https://notmuchmail.org) Emacs/26.2 (x86_64-pc-linux-gnu) Date: Thu, 01 Aug 2019 16:09:00 +0200 Message-ID: <87k1bx2d1f.fsf@devup.no> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --=-=-= Content-Type: text/plain The truncated log files seems to happen for other builds as well, even within the Berlin data center. https://ci.guix.gnu.org/log/n3ra1b8ic6qhfinnhb80mrn7snsqws9d-geocode-glib-3.26.0 https://ci.guix.gnu.org/log/zqhqlib00i8f7f10g4c2dfzprw16h4xv-scintilla-4.2.0 https://ci.guix.gnu.org/log/718jmbq94mvdgnmjyqgxgy7zaj8xzxk3-htslib-1.9 All of these builds are for i686-linux. Mark: are the armhf nodes still operational? I would like to re-enable them again, since we desperately need the computing power with four huge branches going concurrently at the moment. --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAl1C8nwACgkQoqBt8qM6 VPoYkwgAmYCwtCBizPMO0PQZEaX+jcl7LeRl77opAd4zdgNVoaKyukReqyprYgA3 0Uwp2j/Q4wwfVUd8GP5nKn4kdhhRWAK+DdNaOqcOyaBNH2jHhZgWSAnj9UkLwtA0 ebDC3g8AVTlFMGlVQc/21w2c1zpYOxPRIn+AdI04bLuKeAKTzJLDfRF+njE+Hf55 5w9EjyW+K6beR+A0jPLm/6Z9aig8QCxzkboSQdAEECtjMWyKTpDA1tu2YAAw27sC pc0hpJPS/mIdNQv36B5rbJ9TCF4zp9DzQj22BkvUMaVBb3MVax5+qizE6StDFpd1 t9uKVGo5zGnPNO9/MxOj9YzMEHVAlg== =PORt -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 01 11:56:33 2019 Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 15:56:33 +0000 Received: from localhost ([127.0.0.1]:54918 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htDRR-00078q-2I for submit@debbugs.gnu.org; Thu, 01 Aug 2019 11:56:33 -0400 Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21317) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htDRO-00078f-Ge for 36754@debbugs.gnu.org; Thu, 01 Aug 2019 11:56:31 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1564674978; cv=none; d=zoho.com; s=zohoarc; b=eJgExjFecR9RldkOih3G0/gcsdUv9PiE7570fcuI6tgriSoYeuACcZpoqlPBobMT6aiJ+GBkPhG/uTJRtrEaIYKtfz/O/z6i3cG6SZyeiiAFZH7GSW5dfIfAzgsjznlkHSiWAVpupNAzwrbL0S5mG9NILm+q9DF5L7PhdFPM3Jc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564674978; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=LSw/N+WGCJZnGBvEpR+lK3cMxiJYDJPmX8OBGzkyAoE=; b=VUMJx2VUd7X9kmLPbJ2FayAiRylIVxLtZ3JR7+iLpWzb3pNf2iZnVzjFFaq6p6jpu0XxNvj31VRUESvYoRiy3rnBNtm5Cr2b+hQkUIFw+L2/1KFaYoraHeXqm8Z5ACk3QRZyVN8iP3EHT9TzuOHiyipKZ41O1a4yUsfe3Wz06jE= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1564674978; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; l=651; bh=LSw/N+WGCJZnGBvEpR+lK3cMxiJYDJPmX8OBGzkyAoE=; b=VEPYNikrRY9Ogyd+r4PyH7ph0lkVbYcis8ki2Jw7Kcn5jbadPI8rPsFXbXQv0crH CoYlm8L+3A8Qo04jcy58+dyv2Z3DB+0PbMhTRmrqdHE4gZjX40aCpG8r95iytUQl0dD +NExqc3yRp3cknAKDwhdbhdYhqZCtZWfDrPAKCEY= Received: from localhost (141.80.247.250 [141.80.247.250]) by mx.zohomail.com with SMTPS id 1564673962656270.3743885542227; Thu, 1 Aug 2019 08:39:22 -0700 (PDT) References: <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87pnm0bem6.fsf@elephly.net> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-reply-to: <87pnm0bem6.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Thu, 01 Aug 2019 17:39:19 +0200 Message-ID: <87v9vgsxnc.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Mark H Weaver , Marius Bakke , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ricardo Wurmus writes: > Ludovic Court=C3=A8s writes: > >> Currently we have an SSH tunnel set up on berlin to connect to each of >> these machines via overdrive1.guixsd.org. This setup proved to be >> robust in the past (we used it to connect to another build machine), so >> I suspect something=E2=80=99s wrong on =E2=80=9Cyour=E2=80=9D end of the= network. It=E2=80=99s hard to >> tell exactly what, though. > > FWIW by the end of this week we should have the firewall changes > implemented so we can do without the SSH tunnel. The firewall changes have been applied today. --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 01 12:39:59 2019 Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 16:39:59 +0000 Received: from localhost ([127.0.0.1]:55058 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htE7T-00027q-0e for submit@debbugs.gnu.org; Thu, 01 Aug 2019 12:39:59 -0400 Received: from world.peace.net ([64.112.178.59]:38788) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htE7R-00027Z-3j for 36754@debbugs.gnu.org; Thu, 01 Aug 2019 12:39:57 -0400 Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1htE7J-0002Mm-Ol; Thu, 01 Aug 2019 12:39:49 -0400 From: Mark H Weaver To: Marius Bakke Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-Reply-To: <87k1bx2d1f.fsf@devup.no> References: <87o91n3ps0.fsf@netris.org> <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> <87lfwnxyzg.fsf@gnu.org> <87k1bx2d1f.fsf@devup.no> Date: Thu, 01 Aug 2019 12:37:31 -0400 Message-ID: <87pnlokfix.fsf@netris.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Ricardo Wurmus , 36754@debbugs.gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Marius, Marius Bakke wrote: > The truncated log files seems to happen for other builds as well, even > within the Berlin data center. > > https://ci.guix.gnu.org/log/n3ra1b8ic6qhfinnhb80mrn7snsqws9d-geocode-glib-3.26.0 > https://ci.guix.gnu.org/log/zqhqlib00i8f7f10g4c2dfzprw16h4xv-scintilla-4.2.0 > https://ci.guix.gnu.org/log/718jmbq94mvdgnmjyqgxgy7zaj8xzxk3-htslib-1.9 > > All of these builds are for i686-linux. Thanks, that's very useful information. > Mark: are the armhf nodes still operational? I assume so. They all respond to pings anyway, and I haven't touched them since before they were disconnected from Berlin. (I would need to boot up my other, more secure computer to try SSHing into them). > I would like to re-enable them again, since we desperately need the > computing power with four huge branches going concurrently at the > moment. I have no objection, but since Ludovic made the decision to disconnect them, it would be good to hear from him first. Thanks, Mark From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 01 17:06:38 2019 Received: (at 36754) by debbugs.gnu.org; 1 Aug 2019 21:06:38 +0000 Received: from localhost ([127.0.0.1]:55504 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htIHW-0007Na-7y for submit@debbugs.gnu.org; Thu, 01 Aug 2019 17:06:38 -0400 Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21351) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1htIHU-0007NR-2p for 36754@debbugs.gnu.org; Thu, 01 Aug 2019 17:06:36 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1564693585; cv=none; d=zoho.com; s=zohoarc; b=a8VPKkMxkRiJ9i+HLUTdJFUiqr0fQJlyVVHwIlvjYnlCcjwLpAEfV4wDgWqBa3zpcrYPLA5LhYNgZbOlo++/TC4wni9QRx8OHMy3WmaKHLpyC5sKSVEvgUvH2tulhgH17SNxj8HoDGfDXTgJr7T9xghHT8xDxURDki2/gVdUR4o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564693585; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=axLRGfbK+9l7ZwNs/qbpLnyOlYAMBEs09jfQsaPzjtA=; b=ZeYAR6ohta4iegqouOCKu3K6iKKYVYYL/q18hs+L6wMAAv5TCjMMvsnzC5VnODD7C4J/8mI1thBg2yKaEGxnAzZz+8l2DmRoAsD4A2lTnhDr672dBFamlB0oCk0N/JoEZ+SU2IbKaiiQrjqpIu/41J9nonsxqFIT0Ncj6DUehns= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1564693585; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type; l=755; bh=axLRGfbK+9l7ZwNs/qbpLnyOlYAMBEs09jfQsaPzjtA=; b=GjWe6YbUowTGAnCHlVdHKf1rxzCiRrfuZBCLKzv1hboDk4C/7bt+nZh1jvawIWx6 LZdiiwcXhn/0aMrSMIWwJzXRov0e50KwUwKvvr5QOHL1UgCAKVTy4AZWzc1uFQMEWbX jG833oEnVMUDTQJWsCobp+RgsQpJtPq3YLHNn8UY= Received: from localhost (p54AD4FAD.dip0.t-ipconnect.de [84.173.79.173]) by mx.zohomail.com with SMTPS id 1564693583991395.61173015756094; Thu, 1 Aug 2019 14:06:23 -0700 (PDT) References: <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> <87lfwnxyzg.fsf@gnu.org> <87k1bx2d1f.fsf@devup.no> <87pnlokfix.fsf@netris.org> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Mark H Weaver Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-reply-to: <87pnlokfix.fsf@netris.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Thu, 01 Aug 2019 23:06:19 +0200 Message-ID: <87sgqksiic.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Marius Bakke , Ludovic =?utf-8?Q?Court=C3=A8s?= , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Mark H Weaver writes: >> Mark: are the armhf nodes still operational? > > I assume so. They all respond to pings anyway, and I haven't touched > them since before they were disconnected from Berlin. (I would need to > boot up my other, more secure computer to try SSHing into them). > >> I would like to re-enable them again, since we desperately need the >> computing power with four huge branches going concurrently at the >> moment. > > I have no objection, but since Ludovic made the decision to disconnect > them, it would be good to hear from him first. Now that we should be able to SSH to them directly from Berlin we can try connecting and perhaps upgrading the guix-daemon on these machines. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 07 10:31:10 2019 Received: (at 36754) by debbugs.gnu.org; 7 Aug 2019 14:31:10 +0000 Received: from localhost ([127.0.0.1]:39186 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hvMy6-0003VN-2j for submit@debbugs.gnu.org; Wed, 07 Aug 2019 10:31:10 -0400 Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21308) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hvMy2-0003VC-6C for 36754@debbugs.gnu.org; Wed, 07 Aug 2019 10:31:08 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1565188260; cv=none; d=zoho.com; s=zohoarc; b=g2RUtJMULfeNBG+SJC2xP9Jl71X22vdpKB+6iII2ZudiQ5lrkzt4rEgOg2Pdi6l/AE9EgJCG1UBQPg/vnGHeisjEqZ89hYrkbHpd3X1M8jlbZ1mv4OPoCBklIwXQyPiwm6AlXQPYn77Lhiorkt30NgimeDWHt+Yb/qfWByYMoZM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565188260; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=HS9jOI8lx90obYEmPSAwLw4lha5RWs50sYFIcUj7MMw=; b=UZV3a/lJ+y09e+JFWg0pQTs+ien0D+BhMiuthc7mmjv3JWBcQnGFThVr08mgqKwmxVvkpWPChAvF2mae1LiHHZqMcJY3cYOJ7TrMlBvPqAmyChsaaXMpxEInP5okLpc+dP4UQjK72Sm1UjxBIDmdNt+3m1rTdEkTHi9DRTLj5TQ= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1565188260; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; l=1044; bh=HS9jOI8lx90obYEmPSAwLw4lha5RWs50sYFIcUj7MMw=; b=VtmL+vOrgPXrkX0mBqHsQ5jjnnUOxy2e4HraZOQQqkV1HNrJ+8oosByiLk3kQsGZ fMC+BajBfcWfcb9HjKyPdkqudvqdiUueQx3fVZdnDiuynDlxNF9ZzCyeqeVpACkKdQF e+RumAVUufg0zz6L9e/rxFabOiPdbeuXRRgNqkgU= Received: from localhost (141.80.247.250 [141.80.247.250]) by mx.zohomail.com with SMTPS id 1565188258955296.2090554102373; Wed, 7 Aug 2019 07:30:58 -0700 (PDT) References: <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> <87lfwnxyzg.fsf@gnu.org> <87k1bx2d1f.fsf@devup.no> <87pnlokfix.fsf@netris.org> <87sgqksiic.fsf@elephly.net> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Mark H Weaver Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds In-reply-to: <87sgqksiic.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Wed, 07 Aug 2019 16:30:54 +0200 Message-ID: <87d0hhxd29.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 36754 Cc: Marius Bakke , =?utf-8?Q?L?= =?utf-8?Q?udovic_Court=C3=A8s?= , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ricardo Wurmus writes: > Mark H Weaver writes: > >>> Mark: are the armhf nodes still operational? >> >> I assume so. They all respond to pings anyway, and I haven't touched >> them since before they were disconnected from Berlin. (I would need to >> boot up my other, more secure computer to try SSHing into them). >> >>> I would like to re-enable them again, since we desperately need the >>> computing power with four huge branches going concurrently at the >>> moment. >> >> I have no objection, but since Ludovic made the decision to disconnect >> them, it would be good to hear from him first. > > Now that we should be able to SSH to them directly from Berlin we can > try connecting and perhaps upgrading the guix-daemon on these machines. I have removed the SSH tunnel configuration from /etc/guix/machines.scm and re-enabled the machines. Let=E2=80=99s see if this makes any difference. If not we should try to up= grade Guix on these build machines. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 16 06:26:07 2019 Received: (at 36754) by debbugs.gnu.org; 16 Aug 2019 10:26:07 +0000 Received: from localhost ([127.0.0.1]:53167 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hyZQt-0007XV-4A for submit@debbugs.gnu.org; Fri, 16 Aug 2019 06:26:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:55977) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hyZQr-0007X1-Hb for 36754@debbugs.gnu.org; Fri, 16 Aug 2019 06:26:06 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:41255) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hyZQm-00080D-6Q; Fri, 16 Aug 2019 06:26:00 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49202 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hyZQl-0001TS-4n; Fri, 16 Aug 2019 06:25:59 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds References: <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> <87lfwnxyzg.fsf@gnu.org> <87k1bx2d1f.fsf@devup.no> <87pnlokfix.fsf@netris.org> <87sgqksiic.fsf@elephly.net> <87d0hhxd29.fsf@elephly.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 29 Thermidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Fri, 16 Aug 2019 12:25:57 +0200 In-Reply-To: <87d0hhxd29.fsf@elephly.net> (Ricardo Wurmus's message of "Wed, 07 Aug 2019 16:30:54 +0200") Message-ID: <87blwpieyy.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36754 Cc: Mark H Weaver , Marius Bakke , 36754@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Ricardo Wurmus skribis: > Ricardo Wurmus writes: > >> Mark H Weaver writes: >> >>>> Mark: are the armhf nodes still operational? >>> >>> I assume so. They all respond to pings anyway, and I haven't touched >>> them since before they were disconnected from Berlin. (I would need to >>> boot up my other, more secure computer to try SSHing into them). >>> >>>> I would like to re-enable them again, since we desperately need the >>>> computing power with four huge branches going concurrently at the >>>> moment. >>> >>> I have no objection, but since Ludovic made the decision to disconnect >>> them, it would be good to hear from him first. >> >> Now that we should be able to SSH to them directly from Berlin we can >> try connecting and perhaps upgrading the guix-daemon on these machines. > > I have removed the SSH tunnel configuration from /etc/guix/machines.scm > and re-enabled the machines. > > Let=E2=80=99s see if this makes any difference. Is it working well now? > If not we should try to upgrade Guix on these build machines. I think there=E2=80=99s a misunderstanding: these machines used to run a ve= ry old Guix but I installed 1.0 from scratch before migrating them to berlin. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 12 04:41:46 2019 Received: (at 36754-done) by debbugs.gnu.org; 12 Sep 2019 08:41:46 +0000 Received: from localhost ([127.0.0.1]:43032 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i8Kfi-0005eS-65 for submit@debbugs.gnu.org; Thu, 12 Sep 2019 04:41:46 -0400 Received: from eggs.gnu.org ([209.51.188.92]:55474) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i8Kfg-0005eE-5u for 36754-done@debbugs.gnu.org; Thu, 12 Sep 2019 04:41:45 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:58790) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1i8Kfa-0004sm-I4; Thu, 12 Sep 2019 04:41:38 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=38522 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1i8KfX-0007TT-Gy; Thu, 12 Sep 2019 04:41:38 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Ricardo Wurmus Subject: Re: bug#36754: SSH connections to hydra-slave{1, 2, 3} fail during builds References: <87o91n3ps0.fsf@netris.org> <87zhl656xk.fsf@netris.org> <87pnm04s21.fsf@devup.no> <87ef2gr6z3.fsf@netris.org> <878ssor680.fsf@netris.org> <87lfwo1lz0.fsf@gnu.org> <87ef2fpu2a.fsf@netris.org> <87lfwnxyzg.fsf@gnu.org> <87k1bx2d1f.fsf@devup.no> <87pnlokfix.fsf@netris.org> <87sgqksiic.fsf@elephly.net> <87d0hhxd29.fsf@elephly.net> <87blwpieyy.fsf@gnu.org> Date: Thu, 12 Sep 2019 10:41:32 +0200 In-Reply-To: <87blwpieyy.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 16 Aug 2019 12:25:57 +0200") Message-ID: <875zlxrjnn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36754-done Cc: Mark H Weaver , Marius Bakke , 36754-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, AFAICS we no longer have connection issues to hydra-slave{1,2,3}.netris.org so I=E2=80=99m closing this bug. Ludo=E2=80=99. From unknown Sat Jun 14 21:29:01 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 10 Oct 2019 11:24:07 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator