GNU bug report logs - #15368
HTTP client is slow [2.0.9]

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: tracker <at> debbugs.gnu.org
Subject: bug#15368: closed (HTTP client is slow [2.0.9])
Date: Fri, 23 May 2014 20:15:03 +0000

[Message part 1 (text/plain, inline)]

Your message dated Fri, 23 May 2014 22:14:00 +0200
with message-id <87egzks16f.fsf <at> gnu.org>
and subject line Re: bug#15368: HTTP client is slow [2.0.9]
has caused the debbugs.gnu.org bug report #15368,
regarding HTTP client is slow [2.0.9]
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
15368: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15368
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guile <at> gnu.org
Subject: HTTP client is slow [2.0.9]
Date: Fri, 13 Sep 2013 15:41:27 +0200

I just noticed that our HTTP client is very slow.  Consider this:

--8<---------------cut here---------------start------------->8---
(use-modules (web client)
             (rnrs io ports)
             (rnrs bytevectors)
             (srfi srfi-11)
             (ice-9 format))

(define %uri
  "http://ftp.gnu.org/gnu/idutils/idutils-4.6.tar.xz")

(with-fluids ((%default-port-encoding #f))
  (let*-values (((start)
                 (gettimeofday))
                ((p)
                 (let ((s (open-socket-for-uri %uri)))
                   (setvbuf s _IONBF)
                   s))
                ((r h)
                 (http-get %uri
                           #:port p
                           #:streaming? #t
                           #:decode-body? #f))
                ((d len)
                 (let ((b (get-bytevector-all h)))
                   (values b (bytevector-length b)))
                 ;; (let ((b (make-bytevector (* 5 (expt 2 20)))))
                 ;;   (values b
                 ;;           (get-bytevector-n! h b 0 (bytevector-length b))))
                 )
                ((end)
                 (gettimeofday))
                ((throughput)
                 (let ((duration (- (car end) (car start))))
                   (/ (/ len 1024.) duration 1.0))))
    (format #t "~5,1f KiB/s (total: ~5,1f KiB)~%"
            throughput (/ len 1024.))))
--8<---------------cut here---------------end--------------->8---

Here I get a throughput of ~60 KiB/s, vs. ~400 KiB/s for wget and curl.

Looking at the strace output reveals no real difference: they all make
one syscall for each chunk of 1410 bytes.

‘time’ reports that Guile spends 0.2 s. in user and 0.8 s. in system,
both of which are an order of magnitude higher than wget/curl.

Bypassing the custom binary input ports from http.scm and response.scm
doesn’t make any big difference.  Forcing the zero-copy path in
‘scm_c_read’ doesn’t help much either.

Ideas?

Thanks,
Ludo’.

[Message part 3 (message/rfc822, inline)]

From: ludo <at> gnu.org (Ludovic Courtès)
To: 15368-done <at> debbugs.gnu.org
Subject: Re: bug#15368: HTTP client is slow [2.0.9]
Date: Fri, 23 May 2014 22:14:00 +0200

ludo <at> gnu.org (Ludovic Courtès) skribis:

> Here I get a throughput of ~60 KiB/s, vs. ~400 KiB/s for wget and curl.

There’s one little detail I hadn’t even bothered checking:

          ;; Enlarge the receive buffer.
          (setsockopt s SOL_SOCKET SO_RCVBUF (* 12 1024))


Its effect was to *shrink* the receive buffer from 124 KiB (the default
size, per /proc/sys/net/core/rmem_default) to 12 KiB...

Fixed in 0bb3f94, which will be in 2.0.12.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.