From debbugs-submit-bounces@debbugs.gnu.org Mon Aug 19 18:42:06 2024 Received: (at submit) by debbugs.gnu.org; 19 Aug 2024 22:42:06 +0000 Received: from localhost ([127.0.0.1]:59400 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sgB4g-00036e-G8 for submit@debbugs.gnu.org; Mon, 19 Aug 2024 18:42:06 -0400 Received: from lists.gnu.org ([209.51.188.17]:33804) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sgB4e-00036W-91 for submit@debbugs.gnu.org; Mon, 19 Aug 2024 18:42:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sgB3x-0007eg-Np for bug-guix@gnu.org; Mon, 19 Aug 2024 18:41:21 -0400 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sgB3v-0007xV-Ji for bug-guix@gnu.org; Mon, 19 Aug 2024 18:41:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:subject:date:message-id:mime-version: content-transfer-encoding; bh=9bHPFJihN4fwu1Hl37TFuP8GBxAdWGdPF4cq/2vSpIk=; b=Fkigcaq7G0obLYsg4xKgwkD6bQRu4G1URcUBgF6o/06lwVUnI/VX+hxB M/z/G6EckOZ74GTI83DdVfxhRj8Jw+6aRulE9gRXryHMSjnV2EJ4QzK5T UtRuWVeKE8HPidEKPd/SF13aUM16TbQefYpirfS0ZKuJX6pfHFcAgdnRE E=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludovic.courtes@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.10,160,1719871200"; d="scan'208";a="94332163" Received: from 91-160-117-201.subs.proxad.net (HELO ribbon) ([91.160.117.201]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Aug 2024 00:41:14 +0200 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: bug-guix@gnu.org Subject: [cuirass] Failure to write build log leads to build failure X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Quartidi 4 Fructidor an 232 de la =?utf-8?Q?R=C3=A9v?= =?utf-8?Q?olution=2C?= jour de l'Escourgeon X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 20 Aug 2024 00:41:14 +0200 Message-ID: <878qwsuodx.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=192.134.164.104; envelope-from=ludovic.courtes@inria.fr; helo=mail3-relais-sop.national.inria.fr X-Spam_score_int: -8 X-Spam_score: -0.9 X-Spam_bar: / X-Spam_report: (-0.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NORMAL_HTTP_TO_IP=0.001, NUMERIC_HTTP_ADDR=1.242, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) We occasionally see failed builds with truncated logs on ci.guix. These happens in situations where =E2=80=98cuirass remote-worker=E2=80=99 gets EP= IPE as it sends the build log to =E2=80=98remote-server=E2=80=99: --8<---------------cut here---------------start------------->8--- 2024-08-19 19:54:52 @ substituter-started /gnu/store/sv3z77cgg2788hrl87w35b= fmyhkmkv54-libomp-16.0.6.drv substitute 2024-08-19 19:54:52 Downloading http://141.80.167.131/nar/lzip/sv3z77cgg278= 8hrl87w35bfmyhkmkv54-libomp-16.0.6.drv... 2024-08-19 19:54:52=20 2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv 1.8M= iB/s 00:00 | 1KiB transferred 2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv 942K= iB/s 00:00 | 1KiB transferred 2024-08-19 19:54:52=20 2024-08-19 19:54:52 @ substituter-succeeded /gnu/store/sv3z77cgg2788hrl87w3= 5bfmyhkmkv54-libomp-16.0.6.drv 2024-08-19 19:55:04 warning: zlib error in 'gzwrite' while sending log to 1= 41.80.167.131: 0 2024-08-19 19:55:04 error: gdPO1dI1: unexpected error while building '/gnu/= store/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv': #<&compound-exce= ption components: (#<&external-error> #<&origin origin: "fport_write"> #<&m= essage message: "~A"> #<&irritants irritants: ("Broken pipe")> #<&exception= -with-kind-and-args kind: system-error args: ("fport_write" "~A" ("Broken p= ipe") (32))>)> --8<---------------cut here---------------end--------------->8--- The =E2=80=9Cunexpected error while building=E2=80=9D message was added in ba3ea7947ec67e9d9585e40a30d67dbd8115e73c. This suggests that the =E2=80=98catch #t=E2=80=99 around the =E2=80=98send-= log=E2=80=99 call doesn=E2=80=99t have the desired effect; instead, the =E2=80=98guard=E2=80=99 handler is in= voked. It looks like EPIPE comes from the pipe set up by =E2=80=98build-derivation= s&=E2=80=99 and that the underlying =E2=80=98build-derivations=E2=80=99 call keeps writ= ing to even after =E2=80=98send-log=E2=80=99 has closed it. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 22 10:43:19 2024 Received: (at 72722) by debbugs.gnu.org; 22 Aug 2024 14:43:19 +0000 Received: from localhost ([127.0.0.1]:38184 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sh91y-0004gH-Oz for submit@debbugs.gnu.org; Thu, 22 Aug 2024 10:43:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50052) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sh91u-0004fz-R3 for 72722@debbugs.gnu.org; Thu, 22 Aug 2024 10:43:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sh915-0001D9-8l for 72722@debbugs.gnu.org; Thu, 22 Aug 2024 10:42:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=ik7WZkMhi7HBGwOLH3EPM0Ie3i7z4JOF6QyyN5adrSo=; b=kYwkklOh9uobaZsNHgke 3znJjxkevmBydbC4RNSjyHqBKVaiU93X5SOEAxw3mFhVl89B5syRpPK7UHPDi+oNOnWyW8Bg72G6m nNLxOn5KXprE00tZN9Y+krFMBhO4JsIvW9AwMwBfFSIFxqJOj04PS9ypfYi0WN3lpQfFgpQSwx8OQ qImSuEJPnEf8HnbnrLNBxmi/CFTNFyAeND8obRHkGNKtBhRkBmuWprqVkSTmYa4rIr8xpZ2S/SBg6 Sps/tlv3C+f1//rSGfzCkjW9luitbuDEXammeTu8DzhSQ5dOWg81De1RU/u/IGiWE8RpSVPEFhLyw HwAaT26f2nBfhg==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 72722@debbugs.gnu.org Subject: Re: bug#72722: [cuirass] Failure to write build log leads to build failure In-Reply-To: <878qwsuodx.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Tue, 20 Aug 2024 00:41:14 +0200") References: <878qwsuodx.fsf@inria.fr> Date: Thu, 22 Aug 2024 16:42:20 +0200 Message-ID: <87ttfcliur.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 72722 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Ludovic Court=C3=A8s skribis: > We occasionally see failed builds with truncated logs on ci.guix. These > happens in situations where =E2=80=98cuirass remote-worker=E2=80=99 gets = EPIPE as it > sends the build log to =E2=80=98remote-server=E2=80=99: > > 2024-08-19 19:54:52 @ substituter-started /gnu/store/sv3z77cgg2788hrl87w3= 5bfmyhkmkv54-libomp-16.0.6.drv substitute > 2024-08-19 19:54:52 Downloading http://141.80.167.131/nar/lzip/sv3z77cgg2= 788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv... > 2024-08-19 19:54:52=20 > 2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv 1.= 8MiB/s 00:00 | 1KiB transferred > 2024-08-19 19:54:52 ESC[K libomp-16.0.6.drv 94= 2KiB/s 00:00 | 1KiB transferred > 2024-08-19 19:54:52=20 > 2024-08-19 19:54:52 @ substituter-succeeded /gnu/store/sv3z77cgg2788hrl87= w35bfmyhkmkv54-libomp-16.0.6.drv > 2024-08-19 19:55:04 warning: zlib error in 'gzwrite' while sending log to= 141.80.167.131: 0 > 2024-08-19 19:55:04 error: gdPO1dI1: unexpected error while building '/gn= u/store/sv3z77cgg2788hrl87w35bfmyhkmkv54-libomp-16.0.6.drv': #<&compound-ex= ception components: (#<&external-error> #<&origin origin: "fport_write"> #<= &message message: "~A"> #<&irritants irritants: ("Broken pipe")> #<&excepti= on-with-kind-and-args kind: system-error args: ("fport_write" "~A" ("Broken= pipe") (32))>)> But hey, why does =E2=80=98gzwrite=E2=80=99 fail in the first place? I noticed that this usually happened when dumping big logs (several MiBs) very quickly (typically the unpack phase of a large package like LLVM producing lots of data very quickly.) As it turns out, =E2=80=98send-log=E2=80=99 opens its socket with SOCK_NONB= LOCK, and then passes it to zlib, which writes to it in =E2=80=98gzwrite=E2=80=99. B= ut zlib is not equipped to deal with EAGAIN: it just errors out, with =E2=80=98gzwrite= =E2=80=99 returning Z_ERRNO, hence the bug above. I was able to confirm this hypothesis by running: echo '(log-server (version 0))' | nc -l -p 5000 -v | \ (sleep 10; echo starting >&2; wc -c) and then, from a REPL: scheme@(cuirass remote)> (send-log "127.0.0.1" 5000 "foo.drv" (open-input= -file "llvm.log")) 2024-08-22T16:35:37 warning: zlib error in 'gzwrite' while sending log to= 127.0.0.1: -1 : Resource temporarily unavailable $30 =3D #f QED. (Here I used Guile-zlib 0.2.1 with a small modification to =E2=80=98remote.scm=E2=80=99 so it displays the error message after Z_ERRNO= =3D -1.) Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 23 14:48:15 2024 Received: (at 72722-done) by debbugs.gnu.org; 23 Aug 2024 18:48:15 +0000 Received: from localhost ([127.0.0.1]:40115 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1shZKY-0000vt-Th for submit@debbugs.gnu.org; Fri, 23 Aug 2024 14:48:15 -0400 Received: from eggs.gnu.org ([209.51.188.92]:54620) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1shZKU-0000vc-L2 for 72722-done@debbugs.gnu.org; Fri, 23 Aug 2024 14:48:12 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1shZJd-000287-LH for 72722-done@debbugs.gnu.org; Fri, 23 Aug 2024 14:47:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=kb98CCx6YKlDh+Yu1NbtUWcbDNLCov1/skMngJX4n7U=; b=Ghj+7rop1yyHhpV0YDrK eLp/CJIGbuRnb44GefuNsvahZvb1s1W40RyuwBP4C61MRodjqITm5/7KGIiaKTnJHoIzs/ETXb4N6 OQtj69vzgPdldjRPUu8T4BujjLOtnRLhb3yyEC5CSz6Jv8/fw8+lDgztgxeyaa7SRfWlfm3oPdKIr wiGZM73WmXZmHyyiIRmhLcFqDpiLJx/fzdrS6zvfKvKfcyfNf36nqUZxv4CFpOzABqkPcPIdhJDVT OeeXGZJ7V9hWZsnzjJo4rzISFjZkZf4qhbzpTygwOetSTjFzBja7vasnqBB46rdgnXPm4n0PH8Jqp C9GFKMObMqBh+A==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 72722-done@debbugs.gnu.org Subject: Re: bug#72722: [cuirass] Failure to write build log leads to build failure In-Reply-To: <87ttfcliur.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Thu, 22 Aug 2024 16:42:20 +0200") References: <878qwsuodx.fsf@inria.fr> <87ttfcliur.fsf@gnu.org> Date: Fri, 23 Aug 2024 20:46:55 +0200 Message-ID: <87ikvrkrfk.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 72722-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s skribis: > As it turns out, =E2=80=98send-log=E2=80=99 opens its socket with SOCK_NO= NBLOCK, and > then passes it to zlib, which writes to it in =E2=80=98gzwrite=E2=80=99. = But zlib is > not equipped to deal with EAGAIN: it just errors out, with =E2=80=98gzwri= te=E2=80=99 > returning Z_ERRNO, hence the bug above. Fixed in 59010a5ba32a5f0802d28900908ee9c75f473a66. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 23 14:48:38 2024 Received: (at 72722-done) by debbugs.gnu.org; 23 Aug 2024 18:48:38 +0000 Received: from localhost ([127.0.0.1]:40119 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1shZKw-0000wg-8K for submit@debbugs.gnu.org; Fri, 23 Aug 2024 14:48:38 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39890) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1shZKv-0000wQ-63 for 72722-done@debbugs.gnu.org; Fri, 23 Aug 2024 14:48:37 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1shZK4-0002Bm-Dq for 72722-done@debbugs.gnu.org; Fri, 23 Aug 2024 14:47:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=kb98CCx6YKlDh+Yu1NbtUWcbDNLCov1/skMngJX4n7U=; b=gmey0IqGlS8wEokRqtWk T/N6OVPOxcKbecXmeZk7MekGFMO+iX7DAld0WCIvcY/2EPxcpyX51F9XzIEEoVwOToSude1dxAN2A leoSfCHK1KnX3phhfHIsjHDHVvCF5LQ2vIHogVDX1AWjETfp2mau+sQKGTYsALHPNxWX74R8KOHDs OLkri+sytVt7EX5Xw/sDGuOV2GLsnNz44dO9OqAtleeWfvTcS9xAHEPxX4Xmb7lJgLI4/wkojS8mW QGCxxsTPceJ1PTO7Kan8x0E7XSfh5B4owudkzOH/ywZX/+h7R6dK8HK82eh2fooJThn7Z+SXto12g 7y0s/MEckUQIiw==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 72722-done@debbugs.gnu.org Subject: Re: bug#72722: [cuirass] Failure to write build log leads to build failure In-Reply-To: <87ttfcliur.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Thu, 22 Aug 2024 16:42:20 +0200") References: <878qwsuodx.fsf@inria.fr> <87ttfcliur.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Date: Fri, 23 Aug 2024 20:47:40 +0200 Message-ID: <87h6bbkreb.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 72722-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s skribis: > As it turns out, =E2=80=98send-log=E2=80=99 opens its socket with SOCK_NO= NBLOCK, and > then passes it to zlib, which writes to it in =E2=80=98gzwrite=E2=80=99. = But zlib is > not equipped to deal with EAGAIN: it just errors out, with =E2=80=98gzwri= te=E2=80=99 > returning Z_ERRNO, hence the bug above. Fixed in 59010a5ba32a5f0802d28900908ee9c75f473a66. Ludo=E2=80=99. From unknown Tue Jun 17 22:28:15 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 21 Sep 2024 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator