GNU bug report logs - #37044
27.0.50; url-http breaks behind proxy when reusing existing connection

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 37044 in the body.
You can then email your comments to 37044 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Ringo <davidmringo <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.0.50; url-http breaks behind proxy when reusing existing connection
Date: Thu, 15 Aug 2019 19:36:44 -0600

[Message part 1 (text/plain, inline)]

When trying to make more than one HTTPS request to the same host via
`url-retrieve-synchronously' behind my work proxy, only the first request
succeeds.  Depending on the host, subsequent requests result in 400/404/302
and perhaps a couple other response codes.  There's some minimal code below
that can be run with emacs -Q, but I don't know how to replicate the proxy.

It seems that commit 84613dae5c3 is at least partially responsible for
this.  It's a small change to `url-https-proxy-after-change-function' that
binds `url-http-proxy' around the call to `url-http-create-request'.
Without that binding, I see no errors.  I tested this both in 26.2.90 which
does not include that commit, and the revision in this report by simply
redefining the function to the older version.

This happens both when `url-proxy-services' is set manually and when it is
set from environment variables.

I'm not sure if this is a problem with Emacs, the proxy I'm behind, or if
there is some configuration I should be aware of that already addresses
this.  Please let me know if there's any other useful information I can
provide.

Sample code and the corresponding url-http buffer headers follow.  The
specific URLs don't seem to matter as long they are HTTPS.  The optional
arguments also don't seem to make a difference.

- David

Code:
(url-retrieve-synchronously "https://xkcd.com/1/")
(url-retrieve-synchronously "https://xkcd.com/2/")

First request (success):
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; charset=UTF-8
Last-Modified: Thu, 15 Aug 2019 11:24:39 GMT
Expires: Thu, 15 Aug 2019 21:22:11 GMT
Cache-Control: max-age=300
Content-Length: 3012
Accept-Ranges: bytes
Date: Fri, 16 Aug 2019 01:32:01 GMT
Via: 1.1 varnish
Age: 142
Connection: keep-alive
X-Served-By: cache-mdw17348-MDW
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1565919121.298692,VS0,VE0
Vary: Accept-Encoding

Second request (failuer):
HTTP/1.1 400 Bad Request
Server: nginx
Content-Type: text/html; charset=UTF-8
Content-Length: 166
Accept-Ranges: bytes
Date: Fri, 16 Aug 2019 01:32:01 GMT
Via: 1.1 varnish
Connection: keep-alive
X-Served-By: cache-mdw17348-MDW
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1565919121.349321,VS0,VE26


In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu)
 of 2019-08-07 built on slartibartfast
Repository revision: d5622eb6fff94714c5d5a64c98c5e02bc1be478c
Repository branch: master
Windowing system distributor 'Fedora Project', version 11.0.12005000
System Description: Fedora 30 (Workstation Edition)

Recent messages:
Making completion list...
Quit
Note: file is write protected
View mode: type C-h for help, h for commands, q to quit.
previous-line: Beginning of buffer [2 times]
Mark saved where search started
Note: file is write protected
View mode: type C-h for help, C-h m for commands, M-x View-quit to quit.
Quit [3 times]
Mark saved where search started
Quit
Configured using:
 'configure --with-imagemagick --with-modules --with-x-toolkit=no'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF XFT ZLIB OLDXMENU X11 XDBE XIM MODULES THREADS PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Info

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired dired-loaddefs format-spec
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs text-property-search time-date mm-decode mm-bodies
mm-encode mailabbrev gmm-utils mailheader sendmail goto-addr thingatpt
mule-util jka-compr info noutline outline easy-mmode view help-mode
easymenu misearch multi-isearch mail-utils gnutls network-stream
url-cache url-http url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util
ietf-drums mail-prsvr url-gw nsm rmc puny url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map seq byte-opt gv bytecomp byte-compile
cconv cl-loaddefs cl-lib url-vars mailcap tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting x multi-tty make-network-process
emacs)

Memory information:
((conses 16 76232 12727)
 (symbols 48 8198 1)
 (strings 32 28196 1967)
 (string-bytes 1 800056)
 (vectors 16 13427)
 (vector-slots 8 162016 12678)
 (floats 8 43 388)
 (intervals 56 1044 0)
 (buffers 992 19))

[Message part 2 (text/html, inline)]

Message #8 received at 37044 <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: David Ringo <davidmringo <at> gmail.com>
Cc: 37044 <at> debbugs.gnu.org
Subject: Re: bug#37044: 27.0.50; url-http breaks behind proxy when reusing
 existing connection
Date: Fri, 16 Aug 2019 13:05:59 -0400

Hi David,

David Ringo <davidmringo <at> gmail.com> writes:

> It seems that commit 84613dae5c3 is at least partially responsible for
> this.  It's a small change to `url-https-proxy-after-change-function'
> that binds `url-http-proxy' around the call to
> `url-http-create-request'.

This might be addressed by the fix for bug#35969 that I pushed last
night, can you retry with master tip?

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=35969

Thanks,
Thomas

Message #11 received at 37044 <at> debbugs.gnu.org (full text, mbox):

From: David Ringo <davidmringo <at> gmail.com>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: 37044 <at> debbugs.gnu.org
Subject: Re: bug#37044: 27.0.50; url-http breaks behind proxy when reusing
 existing connection
Date: Fri, 16 Aug 2019 18:58:18 -0600

[Message part 1 (text/plain, inline)]

Hi Thomas,

I haven't been able to test on the machine I submitted the report from, but
on some other shared machines behind the proxy, it looks like your fix
seems to work (while the revision I was on is still broken on those
machines). Thanks for all the work on the other bug -- that seemed like a
long process.

I'm not super familiar with debbugs, but if there's anything I should do to
mark this as closed or resolved, please let me know.

- David

On Fri, Aug 16, 2019 at 11:06 AM Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
wrote:

> Hi David,
>
> David Ringo <davidmringo <at> gmail.com> writes:
>
> > It seems that commit 84613dae5c3 is at least partially responsible for
> > this.  It's a small change to `url-https-proxy-after-change-function'
> > that binds `url-http-proxy' around the call to
> > `url-http-create-request'.
>
> This might be addressed by the fix for bug#35969 that I pushed last
> night, can you retry with master tip?
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=35969
>
> Thanks,
> Thomas
>

[Message part 2 (text/html, inline)]

Message #16 received at 37044-done <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: David Ringo <davidmringo <at> gmail.com>
Cc: 37044-done <at> debbugs.gnu.org
Subject: Re: bug#37044: 27.0.50; url-http breaks behind proxy when reusing
 existing connection
Date: Fri, 16 Aug 2019 22:09:07 -0400

Hi David,

David Ringo <davidmringo <at> gmail.com> writes:

> Hi Thomas,
>
> I haven't been able to test on the machine I submitted the report
> from, but on some other shared machines behind the proxy, it looks
> like your fix seems to work (while the revision I was on is still
> broken on those machines). Thanks for all the work on the other bug --
> that seemed like a long process.
>
> I'm not super familiar with debbugs, but if there's anything I should
> do to mark this as closed or resolved, please let me know.

Thanks for the bug report and for confirming the fix.  I'm marking this
"done".

Thomas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.