From unknown Sat Sep 20 09:28:19 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#19939 <19939@debbugs.gnu.org> To: bug#19939 <19939@debbugs.gnu.org> Subject: Status: http client: Chunks shouldn't be read at once Reply-To: bug#19939 <19939@debbugs.gnu.org> Date: Sat, 20 Sep 2025 16:28:19 +0000 retitle 19939 http client: Chunks shouldn't be read at once reassign 19939 guile submitter 19939 ludo@gnu.org (Ludovic Court=C3=A8s) severity 19939 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 24 15:59:22 2015 Received: (at submit) by debbugs.gnu.org; 24 Feb 2015 20:59:22 +0000 Received: from localhost ([127.0.0.1]:57050 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YQMZe-000803-Bq for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:36392) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YQMZc-0007zr-7q for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQMZW-0003WL-9n for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:15 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:51217) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZW-0003WF-7B for submit@debbugs.gnu.org; Tue, 24 Feb 2015 15:59:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38110) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZV-0003E9-7P for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YQMZQ-0003UW-98 for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:13 -0500 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38688) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YQMZQ-0003US-6D for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:08 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:44849 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1YQMZP-0002cH-Lm for bug-guile@gnu.org; Tue, 24 Feb 2015 15:59:08 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: bug-guile@gnu.org Subject: http client: Chunks shouldn't be read at once X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 =?utf-8?Q?Vent=C3=B4se?= an 223 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0xEA52ECF4 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 83C4 F8E5 10A3 3B4C 5BEA D15D 77DD 95E2 EA52 ECF4 X-OS: x86_64-unknown-linux-gnu Date: Tue, 24 Feb 2015 21:59:05 +0100 Message-ID: <87ioer9gbq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) As of 2.0.11, the chunked input port of the HTTP client reads whole chunks at once: --8<---------------cut here---------------start------------->8--- (define (read-chunk port) (let ((size (read-chunk-header port))) (read-chunk-body port size))) (define (read-chunk-body port size) (let ((bv (get-bytevector-n port size))) (get-u8 port) ; CR (get-u8 port) ; LF bv)) (define* (make-chunked-input-port port #:key (keep-alive? #f)) "Returns a new port which translates HTTP chunked transfer encoded data from PORT into a non-encoded format. Returns eof when it has read the final chunk from PORT. This does not necessarily mean that there is no more data on PORT. When the returned port is closed it will also close PORT, unless the KEEP-ALIVE? is true." (define (next-chunk) (read-chunk port)) [...] (define (read! bv idx to-read) [...] (set! buffer (next-chunk)) [...] (make-custom-binary-input-port "chunked input port" read! #f #f close)) --8<---------------cut here---------------end--------------->8--- This is undesirable because: 1. the HTTP server can produce arbitrarily large chunks, leading to large memory use in the client (nginx does indeed produce very large chunks in some cases); 2. it adds an extra level of buffering that the caller of =E2=80=98http-g= et=E2=80=99 does not control (a read of 1 byte from the HTTP body port leads to an actual read of a whole chunk); 3. it introduces extra copying and allocations. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 01 17:44:27 2015 Received: (at 19939-done) by debbugs.gnu.org; 1 Mar 2015 22:44:27 +0000 Received: from localhost ([127.0.0.1]:33599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YSCb5-0001hg-80 for submit@debbugs.gnu.org; Sun, 01 Mar 2015 17:44:27 -0500 Received: from fencepost.gnu.org ([208.118.235.10]:45661 ident=Debian-exim) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YSCb2-0001hY-Rr for 19939-done@debbugs.gnu.org; Sun, 01 Mar 2015 17:44:25 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:59428 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1YSCb2-0006Wh-0R for 19939-done@debbugs.gnu.org; Sun, 01 Mar 2015 17:44:24 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 19939-done@debbugs.gnu.org Subject: Re: bug#19939: http client: Chunks shouldn't be read at once References: <87ioer9gbq.fsf@gnu.org> Date: Sun, 01 Mar 2015 23:44:20 +0100 In-Reply-To: <87ioer9gbq.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 24 Feb 2015 21:59:05 +0100") Message-ID: <87h9u472yj.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 19939-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Fixed in commit 00d3ecf, which will be in 2.0.12. Ludo=E2=80=99. From unknown Sat Sep 20 09:28:19 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 30 Mar 2015 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator