From unknown Sat Jun 21 12:34:51 2025 X-Loop: help-debbugs@gnu.org Subject: bug#38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 28 Nov 2019 09:23:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 38411 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: bug-Guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.157493293215279 (code B ref -1); Thu, 28 Nov 2019 09:23:01 +0000 Received: (at submit) by debbugs.gnu.org; 28 Nov 2019 09:22:12 +0000 Received: from localhost ([127.0.0.1]:56630 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaG03-0003yN-2T for submit@debbugs.gnu.org; Thu, 28 Nov 2019 04:22:12 -0500 Received: from lists.gnu.org ([209.51.188.17]:50991) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaG01-0003yE-Hm for submit@debbugs.gnu.org; Thu, 28 Nov 2019 04:22:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41244) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iaFzy-0002QS-4f for bug-Guix@gnu.org; Thu, 28 Nov 2019 04:22:08 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED,BAYES_50, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39444) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaFzx-0004KF-OE for bug-Guix@gnu.org; Thu, 28 Nov 2019 04:22:05 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37000 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaFzw-0004eF-8C for bug-Guix@gnu.org; Thu, 28 Nov 2019 04:22:04 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 Frimaire an 228 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 28 Nov 2019 10:22:01 +0100 Message-ID: <871rtsz77q.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.8 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.8 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Starting from a couple of days ago (it seems; roughly around the same time berlin hit ENOSPC), people have been experiencing issues during the =E2=80=9Cupdating list of substitutes=E2=80=9D phase from https://ci.guix.g= nu.org, where they=E2=80=99d get an ugly backtrace when they=E2=80=99re at 80% or so. Here=E2=80=99s a small reproducer: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=http-pipelining.scm Content-Description: the reproducer (use-modules (guix scripts substitute) (srfi srfi-1) (srfi srfi-26) (web uri) (web request) (web response) (rnrs io ports)) (define http-multiple-get (@@ (guix scripts substitute) http-multiple-get)) (define %base-url "https://berlin.guix.gnu.org") (define %request-count ;; Number of requests to send. Starts failing at 85 (that is, we don't ;; receive the 85th response). 200) (http-multiple-get (string->uri %base-url) (lambda (request response port result) (let ((len (or (response-content-length response) 0))) (pk 'resp (length result) (uri-path (request-uri request))) (get-bytevector-n port len) (cons result result))) '() (unfold (cut >= <> %request-count) (lambda (n) (build-request (string->uri (string-append %base-url "/" (string-pad (number->string n) 32 #\a) ".narinfo")) #:method 'GET #:headers '((User-Agent . "GNU Guile")))) 1+ 0) #:verify-certificate? #f) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable You=E2=80=99ll see that it hangs waiting for a response as soon as you pipe= line 85 requests or more. Note that: 1. https://bayfront.guix.gnu.org doesn=E2=80=99t have that problem; 2. http://ci.guix.gnu.org doesn=E2=80=99t have that problem; 3. when you send 85 requests, it hangs waiting for the 85th response; but when you send 200 requests, it hangs waiting for the 160th response; so it seems it=E2=80=99s not just a matter of TLS record siz= e. I suspected something having to do with TLS record size limits, but item #3 above may invalidate that hypothesis. The weird thing is that we haven=E2=80=99t change the config of berlin in r= ecent days. Berlin runs nginx 1.17.5 on openssl 1.1.1d, while bayfront runs nginx 1.17.0 on openssl 1.0.2p. I very much welcome any ideas you may have! Thanks, Ludo=E2=80=99. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Nov 28 04:24:46 2019 Received: (at control) by debbugs.gnu.org; 28 Nov 2019 09:24:46 +0000 Received: from localhost ([127.0.0.1]:56635 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaG2Y-00042S-9g for submit@debbugs.gnu.org; Thu, 28 Nov 2019 04:24:46 -0500 Received: from eggs.gnu.org ([209.51.188.92]:49323) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaG2W-00042E-DO for control@debbugs.gnu.org; Thu, 28 Nov 2019 04:24:44 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39463) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaG2P-0002pF-Ki for control@debbugs.gnu.org; Thu, 28 Nov 2019 04:24:39 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37004 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaG2N-0004lb-SC for control@debbugs.gnu.org; Thu, 28 Nov 2019 04:24:36 -0500 Date: Thu, 28 Nov 2019 10:24:33 +0100 Message-Id: <87zhggxsj2.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #38411 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.8 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.8 (-) severity 38411 serious quit From unknown Sat Jun 21 12:34:51 2025 X-Loop: help-debbugs@gnu.org Subject: bug#38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 28 Nov 2019 10:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38411 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 38411@debbugs.gnu.org Received: via spool by 38411-submit@debbugs.gnu.org id=B38411.157493516426196 (code B ref 38411); Thu, 28 Nov 2019 10:00:02 +0000 Received: (at 38411) by debbugs.gnu.org; 28 Nov 2019 09:59:24 +0000 Received: from localhost ([127.0.0.1]:56664 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaGa4-0006oS-6s for submit@debbugs.gnu.org; Thu, 28 Nov 2019 04:59:24 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40713) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaGa0-0006oD-SR for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 04:59:23 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39693) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaGZv-0007Ie-JE for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 04:59:15 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37064 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaGZu-00028I-CD for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 04:59:14 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <871rtsz77q.fsf@gnu.org> Date: Thu, 28 Nov 2019 10:59:12 +0100 In-Reply-To: <871rtsz77q.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Thu, 28 Nov 2019 10:22:01 +0100") Message-ID: <87r21sxqxb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: 0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) > The weird thing is that we haven=E2=80=99t change the config of berlin in= recent > days. Berlin runs nginx 1.17.5 on openssl 1.1.1d, while bayfront runs > nginx 1.17.0 on openssl 1.0.2p. A new data point: nginx 1.17.4 from a month ago works fine: --8<---------------cut here---------------start------------->8--- root@berlin ~/maintenance/hydra# guix system list-generations 2m=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 [=E2=80=A6] Generation 238 Oct 23 2019 18:02:22 file name: /var/guix/profiles/system-238-link canonical file name: /gnu/store/28xk8ny9qslyqgi7pjyz4d2x0xrxpw07-system label: GNU with Linux-Libre 5.3.7 bootloader: grub root device: label: "my-root" kernel: /gnu/store/qpasq1pkzb47w5pjzs80pvslv1n7ja1m-linux-libre-5.3.7/bzI= mage [=E2=80=A6] root@berlin ~/maintenance/hydra# guix gc -R /gnu/store/28xk8ny9qslyqgi7pjyz= 4d2x0xrxpw07-system|grep nginx /gnu/store/4gbpgcr4zc4qf59yq2a008maycfwra4n-shepherd-nginx.scm /gnu/store/3gqi5cahcwjfvv0bbfqv8ifir2vrqirh-nginx.conf /gnu/store/s9fm4d5ii8bnh9zv5k78mzjvcl3dipbh-shepherd-nginx.go /gnu/store/zj3mxk3r2dka56favm357kmgywnv5imk-nginx-1.17.4 root@berlin ~/maintenance/hydra# guix gc --references /gnu/store/zj3mxk3r2d= ka56favm357kmgywnv5imk-nginx-1.17.4/sbin/nginx |grep ssl /gnu/store/1dvkm6b97667qd36rsnw4g6isnsmpym7-openssl-1.1.1d --8<---------------cut here---------------end--------------->8--- So for now berlin is running this nginx. I=E2=80=99ll try with 1.17.6 and 1.17.4 on top of current master. Ludo=E2=80=99. From unknown Sat Jun 21 12:34:51 2025 X-Loop: help-debbugs@gnu.org Subject: bug#38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 28 Nov 2019 10:05:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38411 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 38411@debbugs.gnu.org Received: via spool by 38411-submit@debbugs.gnu.org id=B38411.15749354631773 (code B ref 38411); Thu, 28 Nov 2019 10:05:02 +0000 Received: (at 38411) by debbugs.gnu.org; 28 Nov 2019 10:04:23 +0000 Received: from localhost ([127.0.0.1]:56668 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaGes-0000SV-Qh for submit@debbugs.gnu.org; Thu, 28 Nov 2019 05:04:23 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42441) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaGeq-0000SC-5L for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:04:20 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39777) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaGei-0004RF-U4 for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:04:13 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37068 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaGeh-0008KL-TN for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:04:12 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <871rtsz77q.fsf@gnu.org> Date: Thu, 28 Nov 2019 11:04:09 +0100 In-Reply-To: <871rtsz77q.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Thu, 28 Nov 2019 10:22:01 +0100") Message-ID: <87k17kxqp2.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.8 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.8 (-) Ludovic Court=C3=A8s skribis: > You=E2=80=99ll see that it hangs waiting for a response as soon as you pi= peline > 85 requests or more. When you look at /var/log/guix-publish.log, you see that nginx did not forward the 85th request (and beyond) to =E2=80=98guix publish=E2=80=99. Interestingly, if you C-c the client while it=E2=80=99s waiting for further responses, nginx suddenly forwards all the pending requests to =E2=80=98guix publish=E2=80=99. Ludo=E2=80=99. From unknown Sat Jun 21 12:34:51 2025 X-Loop: help-debbugs@gnu.org Subject: bug#38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 28 Nov 2019 10:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38411 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 38411@debbugs.gnu.org Received: via spool by 38411-submit@debbugs.gnu.org id=B38411.15749363113152 (code B ref 38411); Thu, 28 Nov 2019 10:19:01 +0000 Received: (at 38411) by debbugs.gnu.org; 28 Nov 2019 10:18:31 +0000 Received: from localhost ([127.0.0.1]:56693 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaGsZ-0000ol-AG for submit@debbugs.gnu.org; Thu, 28 Nov 2019 05:18:31 -0500 Received: from eggs.gnu.org ([209.51.188.92]:43755) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaGsX-0000oY-27 for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:18:29 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39892) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaGsQ-0005UR-SW for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:18:22 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37120 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaGsQ-0000r5-F8 for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:18:22 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <871rtsz77q.fsf@gnu.org> <87r21sxqxb.fsf@gnu.org> Date: Thu, 28 Nov 2019 11:18:19 +0100 In-Reply-To: <87r21sxqxb.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Thu, 28 Nov 2019 10:59:12 +0100") Message-ID: <87fti8xq1g.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: 0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) nginx 1.17.6 is similarly broken: --8<---------------cut here---------------start------------->8--- root@berlin ~/maintenance/hydra# guix describe Generation 45 Nov 28 2019 11:02:51 (current) guix 18a5575 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 18a5575ec530f0e7a2e27f2aa3b5addf20da0f87 root@berlin ~/maintenance/hydra# guix build nginx /gnu/store/6q44kjf59rgkvn0ip8m0454ybszhjpy0-nginx-1.17.6 root@berlin ~/maintenance/hydra# guix gc --references $(guix build nginx)|grep ssl /gnu/store/1dvkm6b97667qd36rsnw4g6isnsmpym7-openssl-1.1.1d --8<---------------cut here---------------end--------------->8--- Ludo'. From unknown Sat Jun 21 12:34:51 2025 X-Loop: help-debbugs@gnu.org Subject: bug#38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 28 Nov 2019 10:30:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38411 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 38411@debbugs.gnu.org Received: via spool by 38411-submit@debbugs.gnu.org id=B38411.15749369624132 (code B ref 38411); Thu, 28 Nov 2019 10:30:02 +0000 Received: (at 38411) by debbugs.gnu.org; 28 Nov 2019 10:29:22 +0000 Received: from localhost ([127.0.0.1]:56701 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaH33-00014Z-Nh for submit@debbugs.gnu.org; Thu, 28 Nov 2019 05:29:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56887) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaH32-00014O-KO for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:29:20 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39963) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaH2v-0003f3-Ei for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:29:13 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37148 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaH2s-0004q9-Hn for 38411@debbugs.gnu.org; Thu, 28 Nov 2019 05:29:12 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <871rtsz77q.fsf@gnu.org> <87r21sxqxb.fsf@gnu.org> Date: Thu, 28 Nov 2019 11:29:08 +0100 In-Reply-To: <87r21sxqxb.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Thu, 28 Nov 2019 10:59:12 +0100") Message-ID: <87a78gxpjf.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.8 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.8 (-) AFAICS everything is working fine with this config: --8<---------------cut here---------------start------------->8--- root@berlin ~/maintenance/hydra# guix describe Generation 45 Nov 28 2019 11:02:51 (current) guix 18a5575 repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 18a5575ec530f0e7a2e27f2aa3b5addf20da0f87 root@berlin ~/maintenance/hydra# git log |head commit 9fdb990982006a4d0ddb68aa71351cc76ef50cdc Author: Ludovic Court=C3=A8s Date: Thu Nov 28 11:23:29 2019 +0100 nginx: berlin: Use nginx 1.17.4. =20=20=20=20 * hydra/nginx/berlin.scm (nginx-1.17.4): New variable. (%nginx-configuration)[nginx]: New field. --8<---------------cut here---------------end--------------->8--- Please let me know if you encounter any issues! I=E2=80=99ll leave it at that for now, but we=E2=80=99ll have to pay attent= ion to that in future upgrades. (Ideally we=E2=80=99d report a bug to nginx but that= =E2=80=99s tricky enough that this alone would take much more time than I can allocate to it.) Ludo=E2=80=99. From unknown Sat Jun 21 12:34:51 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Subject: bug#38411: closed (HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org) Message-ID: References: <871rtsz77q.fsf@gnu.org> X-Gnu-PR-Message: they-closed 38411 X-Gnu-PR-Package: guix Reply-To: 38411@debbugs.gnu.org Date: Wed, 25 Dec 2019 16:52:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1577292722-17244-1" This is a multi-part message in MIME format... ------------=_1577292722-17244-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.= org which was filed against the guix package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 38411@debbugs.gnu.org. --=20 38411: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D38411 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1577292722-17244-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 38411-done) by debbugs.gnu.org; 25 Dec 2019 16:51:16 +0000 Received: from localhost ([127.0.0.1]:54650 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ik9sO-0004T7-Qw for submit@debbugs.gnu.org; Wed, 25 Dec 2019 11:51:16 -0500 Received: from mail-ed1-f43.google.com ([209.85.208.43]:41211) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ik9sK-0004Sk-4d for 38411-done@debbugs.gnu.org; Wed, 25 Dec 2019 11:51:12 -0500 Received: by mail-ed1-f43.google.com with SMTP id c26so20345279eds.8 for <38411-done@debbugs.gnu.org>; Wed, 25 Dec 2019 08:51:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=lnYbPWeIj+hS1AZJrtTykBkN8T5plQa0cOPZB8C5Wcw=; b=SvKkmu2J/3Bq52Ir8Npb0ru7M3I0RED2f/WJskQTQm7kGwVkwOlA8lB3CgFFAB0NTO y4iYcYrLSw3ZefeMYqcpcLKKKgP0RkHnUWYYSsfHGud+B98VEGRliKaC7LtJUa9XIP0x FcmjQj0Ffv+OL0bQ+PQ1SOb/esMUsvDcnniBiGVOO7r2PBOkc/aokr5suz/Oawd66jGy khK8C34Io34AGpt4u0GtWZ9DHk1fBXS/rpJ5Qqu9RjyyoMxMH7C/9HifVQyuU0XmEdXx z/xMtKpaq1Y/NZMEcWeGoowHLYBSef8w+xOeHsb+hq+SIUC4yHT2nEgCc6enfxVOvRdH rNIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=lnYbPWeIj+hS1AZJrtTykBkN8T5plQa0cOPZB8C5Wcw=; b=ACySpopbqKS0oAehNY47xWw55k19oewsij/z73LrebvSSuk5uBiLTn/R27LiAKPo1V FxmgFMNvJJct+2ZH2JYS9MVZj+OqkRvKgCREQPFhnvkMi4ijJLi7wMSk6E0u6sUPLczI W3WUzanQndlREsBmvRrjhxhL3UvojXYKJJ8g4HprZcQCbEZYojOHyAjbdPP0JnEuZsoP 8h8mgZ2+gJxMe8FgLkzVdLhNISJbEDo+HvKtCtzBDmBpfinDz6yemaRgIDe/Zs/893wN A2e/ZNd/YdyKvB4LzKmPE9P6DEvGW1KoYiyfGlMi2qhzNyrhQadgmlJzbE21bn76qAaP nqsQ== X-Gm-Message-State: APjAAAU0sSkGrG2f0N7XlHc3rX/Pxm4LzETn/WS5+FD4976vFHAxOiew +M84sUGykDajDrtKy+r7JTo2doMmETpm10PBxivNRZRwhg== X-Google-Smtp-Source: APXvYqzPYggI6TzUF/G1I23TH44W0PEAZVTluoyYhVkkYtZM1UxijDZXVq0gbccnxACmmU14hkqHivj3H+KYSK6H66I= X-Received: by 2002:a17:906:2894:: with SMTP id o20mr44124783ejd.199.1577292661653; Wed, 25 Dec 2019 08:51:01 -0800 (PST) MIME-Version: 1.0 From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Wed, 25 Dec 2019 17:50:50 +0100 Message-ID: Subject: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org To: 38411-done@debbugs.gnu.org Content-Type: text/plain; charset="UTF-8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 38411-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) This was an upstream regression introduced in nginx 1.17.5. It is fixed in 1.17.7. Fixed by updating nginx to 1.17.7 in commit: 32dfde905229e593f9fe60795d2490f99c27aad5 and updating berlin config in maintenance on commit: 87d451e9c3381b21e6c7208372576abed84df1e6. This is mentioned on then nginx 1.17.7 release notes as: Bugfix: a timeout might occur while handling pipelined requests in an SSL connection; the bug had appeared in 1.17.5. g_bor -- OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21 ------------=_1577292722-17244-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 28 Nov 2019 09:22:12 +0000 Received: from localhost ([127.0.0.1]:56630 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaG03-0003yN-2T for submit@debbugs.gnu.org; Thu, 28 Nov 2019 04:22:12 -0500 Received: from lists.gnu.org ([209.51.188.17]:50991) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iaG01-0003yE-Hm for submit@debbugs.gnu.org; Thu, 28 Nov 2019 04:22:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41244) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iaFzy-0002QS-4f for bug-Guix@gnu.org; Thu, 28 Nov 2019 04:22:08 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.2 required=5.0 tests=ALL_TRUSTED,BAYES_50, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:470:142:3::e]:39444) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1iaFzx-0004KF-OE for bug-Guix@gnu.org; Thu, 28 Nov 2019 04:22:05 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=37000 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1iaFzw-0004eF-8C for bug-Guix@gnu.org; Thu, 28 Nov 2019 04:22:04 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: bug-Guix@gnu.org Subject: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 8 Frimaire an 228 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 28 Nov 2019 10:22:01 +0100 Message-ID: <871rtsz77q.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.8 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.8 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Starting from a couple of days ago (it seems; roughly around the same time berlin hit ENOSPC), people have been experiencing issues during the =E2=80=9Cupdating list of substitutes=E2=80=9D phase from https://ci.guix.g= nu.org, where they=E2=80=99d get an ugly backtrace when they=E2=80=99re at 80% or so. Here=E2=80=99s a small reproducer: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=http-pipelining.scm Content-Description: the reproducer (use-modules (guix scripts substitute) (srfi srfi-1) (srfi srfi-26) (web uri) (web request) (web response) (rnrs io ports)) (define http-multiple-get (@@ (guix scripts substitute) http-multiple-get)) (define %base-url "https://berlin.guix.gnu.org") (define %request-count ;; Number of requests to send. Starts failing at 85 (that is, we don't ;; receive the 85th response). 200) (http-multiple-get (string->uri %base-url) (lambda (request response port result) (let ((len (or (response-content-length response) 0))) (pk 'resp (length result) (uri-path (request-uri request))) (get-bytevector-n port len) (cons result result))) '() (unfold (cut >= <> %request-count) (lambda (n) (build-request (string->uri (string-append %base-url "/" (string-pad (number->string n) 32 #\a) ".narinfo")) #:method 'GET #:headers '((User-Agent . "GNU Guile")))) 1+ 0) #:verify-certificate? #f) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable You=E2=80=99ll see that it hangs waiting for a response as soon as you pipe= line 85 requests or more. Note that: 1. https://bayfront.guix.gnu.org doesn=E2=80=99t have that problem; 2. http://ci.guix.gnu.org doesn=E2=80=99t have that problem; 3. when you send 85 requests, it hangs waiting for the 85th response; but when you send 200 requests, it hangs waiting for the 160th response; so it seems it=E2=80=99s not just a matter of TLS record siz= e. I suspected something having to do with TLS record size limits, but item #3 above may invalidate that hypothesis. The weird thing is that we haven=E2=80=99t change the config of berlin in r= ecent days. Berlin runs nginx 1.17.5 on openssl 1.1.1d, while bayfront runs nginx 1.17.0 on openssl 1.0.2p. I very much welcome any ideas you may have! Thanks, Ludo=E2=80=99. --=-=-=-- ------------=_1577292722-17244-1-- From unknown Sat Jun 21 12:34:51 2025 X-Loop: help-debbugs@gnu.org Subject: bug#38411: closed (HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org) Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 26 Dec 2019 18:15:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 38411 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: =?UTF-8?Q?G=C3=A1bor?= Boskovits Cc: 38411@debbugs.gnu.org Received: via spool by 38411-submit@debbugs.gnu.org id=B38411.157738407731125 (code B ref 38411); Thu, 26 Dec 2019 18:15:01 +0000 Received: (at 38411) by debbugs.gnu.org; 26 Dec 2019 18:14:37 +0000 Received: from localhost ([127.0.0.1]:56225 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ikXee-00085x-UF for submit@debbugs.gnu.org; Thu, 26 Dec 2019 13:14:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56055) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ikXed-00085j-QD for 38411@debbugs.gnu.org; Thu, 26 Dec 2019 13:14:36 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:46913) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ikXeY-0003t1-NP; Thu, 26 Dec 2019 13:14:30 -0500 Received: from [2a01:e35:2ffd:930:f25f:2121:7012:6c8e] (port=57288 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ikXeX-00041i-N7; Thu, 26 Dec 2019 13:14:30 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <871rtsz77q.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 =?UTF-8?Q?Niv=C3=B4se?= an 228 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 26 Dec 2019 19:14:28 +0100 In-Reply-To: (GNU bug Tracking System's message of "Wed, 25 Dec 2019 16:52:02 +0000") Message-ID: <87sgl7gdiz.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi G=C3=A1bor, help-debbugs@gnu.org (GNU bug Tracking System) skribis: > This was an upstream regression introduced in nginx 1.17.5. > It is fixed in 1.17.7. Fixed by updating nginx to 1.17.7 in commit: > 32dfde905229e593f9fe60795d2490f99c27aad5 > and updating berlin config in maintenance on commit: > 87d451e9c3381b21e6c7208372576abed84df1e6. > > This is mentioned on then nginx 1.17.7 release notes as: > > Bugfix: a timeout might occur while handling pipelined requests in an > SSL connection; the bug had appeared in 1.17.5. I=E2=80=99m really happy you were able to pinpoint the issue and to help upstream diagnose it. The power of free software! Thank you, Ludo=E2=80=99.