GNU bug report logs - #19939
http client: Chunks shouldn't be read at once

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: ludo <at> gnu.org (Ludovic Courtès)
Subject: bug#19939: closed (Re: bug#19939: http client: Chunks shouldn't
 be read at once)
Date: Sun, 01 Mar 2015 22:45:03 +0000

[Message part 1 (text/plain, inline)]

Your bug report

#19939: http client: Chunks shouldn't be read at once

which was filed against the guile package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to 19939 <at> debbugs.gnu.org.

-- 
19939: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19939
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: ludo <at> gnu.org (Ludovic Courtès)
To: 19939-done <at> debbugs.gnu.org
Subject: Re: bug#19939: http client: Chunks shouldn't be read at once
Date: Sun, 01 Mar 2015 23:44:20 +0100

Fixed in commit 00d3ecf, which will be in 2.0.12.

Ludo’.

[Message part 3 (message/rfc822, inline)]

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guile <at> gnu.org
Subject: http client: Chunks shouldn't be read at once
Date: Tue, 24 Feb 2015 21:59:05 +0100

As of 2.0.11, the chunked input port of the HTTP client reads whole
chunks at once:

--8<---------------cut here---------------start------------->8---
(define (read-chunk port)
  (let ((size (read-chunk-header port)))
    (read-chunk-body port size)))

(define (read-chunk-body port size)
  (let ((bv (get-bytevector-n port size)))
    (get-u8 port)                       ; CR
    (get-u8 port)                       ; LF
    bv))

(define* (make-chunked-input-port port #:key (keep-alive? #f))
  "Returns a new port which translates HTTP chunked transfer encoded
data from PORT into a non-encoded format. Returns eof when it has
read the final chunk from PORT. This does not necessarily mean
that there is no more data on PORT. When the returned port is
closed it will also close PORT, unless the KEEP-ALIVE? is true."
  (define (next-chunk)
    (read-chunk port))

[...]

  (define (read! bv idx to-read)

[...]

               (set! buffer (next-chunk))

[...]

  (make-custom-binary-input-port "chunked input port" read! #f #f close))
--8<---------------cut here---------------end--------------->8---

This is undesirable because:

  1. the HTTP server can produce arbitrarily large chunks, leading to
     large memory use in the client (nginx does indeed produce very
     large chunks in some cases);

  2. it adds an extra level of buffering that the caller of ‘http-get’
     does not control (a read of 1 byte from the HTTP body port leads
     to an actual read of a whole chunk);

  3. it introduces extra copying and allocations.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.