From unknown Sat Sep 20 09:29:45 2025 X-Loop: help-debbugs@gnu.org Subject: bug#19939: http client: Chunks shouldn't be read at once Resent-From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Tue, 24 Feb 2015 21:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 19939 X-GNU-PR-Package: guile X-GNU-PR-Keywords: To: 19939@debbugs.gnu.org X-Debbugs-Original-To: bug-guile@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.142481156230759 (code B ref -1); Tue, 24 Feb 2015 21:00:02 +0000 Received: (at submit) by debbugs.gnu.org; 24 Feb 2015 20:59:22 +0000 Received: from localhost ([127.0.0.1]:57050 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YQMZe-000803-Bq for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:36392) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YQMZc-0007zr-7q for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQMZW-0003WL-9n for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:15 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:51217) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZW-0003WF-7B for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZV-0003E9-7P for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQMZQ-0003UW-98 for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:13 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38688) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZQ-0003US-6D for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:08 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:44849 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1YQMZP-0002cH-Lm for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:08 -0500 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 =?UTF-8?Q?Vent=C3=B4se?= an 223 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0xEA52ECF4 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 83C4 F8E5 10A3 3B4C 5BEA D15D 77DD 95E2 EA52 ECF4 X-OS: x86_64-unknown-linux-gnu Date: Tue, 24 Feb 2015 21:59:05 +0100 Message-ID: <87ioer9gbq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) As of 2.0.11, the chunked input port of the HTTP client reads whole chunks at once: --8<---------------cut here---------------start------------->8--- (define (read-chunk port) (let ((size (read-chunk-header port))) (read-chunk-body port size))) (define (read-chunk-body port size) (let ((bv (get-bytevector-n port size))) (get-u8 port) ; CR (get-u8 port) ; LF bv)) (define* (make-chunked-input-port port #:key (keep-alive? #f)) "Returns a new port which translates HTTP chunked transfer encoded data from PORT into a non-encoded format. Returns eof when it has read the final chunk from PORT. This does not necessarily mean that there is no more data on PORT. When the returned port is closed it will also close PORT, unless the KEEP-ALIVE? is true." (define (next-chunk) (read-chunk port)) [...] (define (read! bv idx to-read) [...] (set! buffer (next-chunk)) [...] (make-custom-binary-input-port "chunked input port" read! #f #f close)) --8<---------------cut here---------------end--------------->8--- This is undesirable because: 1. the HTTP server can produce arbitrarily large chunks, leading to large memory use in the client (nginx does indeed produce very large chunks in some cases); 2. it adds an extra level of buffering that the caller of =E2=80=98http-g= et=E2=80=99 does not control (a read of 1 byte from the HTTP body port leads to an actual read of a whole chunk); 3. it introduces extra copying and allocations. Ludo=E2=80=99. From unknown Sat Sep 20 09:29:45 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.503 (Entity 5.503) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#19939: closed (Re: bug#19939: http client: Chunks shouldn't be read at once) Message-ID: References: <87h9u472yj.fsf@gnu.org> <87ioer9gbq.fsf@gnu.org> X-Gnu-PR-Message: they-closed 19939 X-Gnu-PR-Package: guile Reply-To: 19939@debbugs.gnu.org Date: Sun, 01 Mar 2015 22:45:03 +0000 Content-Type: multipart/mixed; boundary="----------=_1425249903-6609-1" This is a multi-part message in MIME format... ------------=_1425249903-6609-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #19939: http client: Chunks shouldn't be read at once which was filed against the guile package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 19939@debbugs.gnu.org. --=20 19939: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D19939 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1425249903-6609-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 19939-done) by debbugs.gnu.org; 1 Mar 2015 22:44:27 +0000 Received: from localhost ([127.0.0.1]:33599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YSCb5-0001hg-80 for submit@debbugs.gnu.org; Sun, 01 Mar 2015 17:44:27 -0500 Received: from fencepost.gnu.org ([208.118.235.10]:45661 ident=Debian-exim) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YSCb2-0001hY-Rr for 19939-done@debbugs.gnu.org; Sun, 01 Mar 2015 17:44:25 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:59428 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1YSCb2-0006Wh-0R for 19939-done@debbugs.gnu.org; Sun, 01 Mar 2015 17:44:24 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 19939-done@debbugs.gnu.org Subject: Re: bug#19939: http client: Chunks shouldn't be read at once References: <87ioer9gbq.fsf@gnu.org> Date: Sun, 01 Mar 2015 23:44:20 +0100 In-Reply-To: <87ioer9gbq.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 24 Feb 2015 21:59:05 +0100") Message-ID: <87h9u472yj.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 19939-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Fixed in commit 00d3ecf, which will be in 2.0.12. Ludo=E2=80=99. ------------=_1425249903-6609-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 24 Feb 2015 20:59:22 +0000 Received: from localhost ([127.0.0.1]:57050 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YQMZe-000803-Bq for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:36392) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YQMZc-0007zr-7q for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQMZW-0003WL-9n for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:15 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:51217) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZW-0003WF-7B for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZV-0003E9-7P for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQMZQ-0003UW-98 for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:13 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38688) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZQ-0003US-6D for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:08 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:44849 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1YQMZP-0002cH-Lm for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:08 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: bug-guile@gnu.org Subject: http client: Chunks shouldn't be read at once X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 =?utf-8?Q?Vent=C3=B4se?= an 223 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0xEA52ECF4 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 83C4 F8E5 10A3 3B4C 5BEA D15D 77DD 95E2 EA52 ECF4 X-OS: x86_64-unknown-linux-gnu Date: Tue, 24 Feb 2015 21:59:05 +0100 Message-ID: <87ioer9gbq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) As of 2.0.11, the chunked input port of the HTTP client reads whole chunks at once: --8<---------------cut here---------------start------------->8--- (define (read-chunk port) (let ((size (read-chunk-header port))) (read-chunk-body port size))) (define (read-chunk-body port size) (let ((bv (get-bytevector-n port size))) (get-u8 port) ; CR (get-u8 port) ; LF bv)) (define* (make-chunked-input-port port #:key (keep-alive? #f)) "Returns a new port which translates HTTP chunked transfer encoded data from PORT into a non-encoded format. Returns eof when it has read the final chunk from PORT. This does not necessarily mean that there is no more data on PORT. When the returned port is closed it will also close PORT, unless the KEEP-ALIVE? is true." (define (next-chunk) (read-chunk port)) [...] (define (read! bv idx to-read) [...] (set! buffer (next-chunk)) [...] (make-custom-binary-input-port "chunked input port" read! #f #f close)) --8<---------------cut here---------------end--------------->8--- This is undesirable because: 1. the HTTP server can produce arbitrarily large chunks, leading to large memory use in the client (nginx does indeed produce very large chunks in some cases); 2. it adds an extra level of buffering that the caller of =E2=80=98http-g= et=E2=80=99 does not control (a read of 1 byte from the HTTP body port leads to an actual read of a whole chunk); 3. it introduces extra copying and allocations. Ludo=E2=80=99. ------------=_1425249903-6609-1--