GNU bug report logs - #38411
HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Ludovic Courtès <ludo <at> gnu.org>
Subject: bug#38411: closed (HTTP pipelining of narinfo requests broken for
 https://ci.guix.gnu.org)
Date: Wed, 25 Dec 2019 16:52:02 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#38411: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 38411 <at> debbugs.gnu.org.

-- 
38411: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=38411
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Gábor Boskovits <boskovits <at> gmail.com>
To: 38411-done <at> debbugs.gnu.org
Subject: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org
Date: Wed, 25 Dec 2019 17:50:50 +0100

This was an upstream regression introduced in nginx 1.17.5.
It is fixed in 1.17.7. Fixed by updating nginx to 1.17.7 in commit:
32dfde905229e593f9fe60795d2490f99c27aad5
and updating berlin config in maintenance on commit:
87d451e9c3381b21e6c7208372576abed84df1e6.

This is mentioned on then nginx 1.17.7 release notes as:

Bugfix: a timeout might occur while handling pipelined requests in an
SSL connection; the bug had appeared in 1.17.5.

g_bor
-- 
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21

[Message part 3 (message/rfc822, inline)]

From: Ludovic Courtès <ludo <at> gnu.org>
To: bug-Guix <at> gnu.org
Subject: HTTP pipelining of narinfo requests broken for https://ci.guix.gnu.org
Date: Thu, 28 Nov 2019 10:22:01 +0100

[Message part 4 (text/plain, inline)]

Starting from a couple of days ago (it seems; roughly around the same
time berlin hit ENOSPC), people have been experiencing issues during the
“updating list of substitutes” phase from https://ci.guix.gnu.org, where
they’d get an ugly backtrace when they’re at 80% or so.

Here’s a small reproducer:

[http-pipelining.scm (text/plain, inline)]

(use-modules (guix scripts substitute)
             (srfi srfi-1)
             (srfi srfi-26)
             (web uri)
             (web request)
             (web response)
             (rnrs io ports))

(define http-multiple-get
  (@@ (guix scripts substitute) http-multiple-get))

(define %base-url
  "https://berlin.guix.gnu.org")

(define %request-count
  ;; Number of requests to send.  Starts failing at 85 (that is, we don't
  ;; receive the 85th response).
  200)

(http-multiple-get (string->uri %base-url)
                   (lambda (request response port result)
                     (let ((len (or (response-content-length response)
                                    0)))
                       (pk 'resp (length result)
                           (uri-path (request-uri request)))
                       (get-bytevector-n port len)
                       (cons result result)))
                   '()
                   (unfold (cut >= <> %request-count)
                           (lambda (n)
                             (build-request
                              (string->uri
                               (string-append
                                %base-url
                                "/"
                                (string-pad (number->string n) 32 #\a)
                                ".narinfo"))
                              #:method 'GET
                              #:headers '((User-Agent . "GNU Guile"))))
                           1+
                           0)
                   #:verify-certificate? #f)

[Message part 6 (text/plain, inline)]

You’ll see that it hangs waiting for a response as soon as you pipeline
85 requests or more.

Note that:

  1. https://bayfront.guix.gnu.org doesn’t have that problem;

  2. http://ci.guix.gnu.org doesn’t have that problem;

  3. when you send 85 requests, it hangs waiting for the 85th response;
     but when you send 200 requests, it hangs waiting for the 160th
     response; so it seems it’s not just a matter of TLS record size.

I suspected something having to do with TLS record size limits, but item
#3 above may invalidate that hypothesis.

The weird thing is that we haven’t change the config of berlin in recent
days.  Berlin runs nginx 1.17.5 on openssl 1.1.1d, while bayfront runs
nginx 1.17.0 on openssl 1.0.2p.

I very much welcome any ideas you may have!

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.