GNU bug report logs - #22744
24.5; url-retrieve callback is not invoked when http response content is empty

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22744 in the body.
You can then email your comments to 22744 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Shiyao Ma <i <at> introo.me>
To: bug-gnu-emacs <at> gnu.org, wmperry <at> gnu.org
Subject: 24.5; url-retrieve callback is not invoked when http response content
 is empty
Date: Sat, 20 Feb 2016 13:09:25 +0800

[Message part 1 (text/plain, inline)]

A working snippet is here:

https://bpaste.net/show/d2bd28c10f46

The callback paste-callback-bpaste is not invoked.

Turning on `url-debug', we can observe that the post response is a mere
http header with status 302, and no *data content*.


Regards.



**************************



In GNU Emacs 24.5.1 (x86_64-apple-darwin14.4.0, NS apple-appkit-1348.17)
 of 2015-08-11 on Mango
Configured using:
 `configure --prefix=/usr/local/Cellar/emacs/24.5
 --enable-locallisppath=/usr/local/share/emacs/site-lisp
 --infodir=/usr/local/Cellar/emacs/24.5/share/info/emacs --without-dbus
 --without-gnutls --with-ns --disable-ns-self-contained'

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  elisp-slime-nav-mode: t
  eldoc-mode: t
  which-key-mode: t
  async-bytecomp-package-mode: t
  evil-jumper-mode: t
  evil-leader-mode: t
  evil-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  evil-local-mode: t
  override-global-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
ad-handle-definition: `evil-mode' got redefined
Source file
`/Users/Eddie/.emacs.d/lisp/bundle/dpaste_de-20131015.525/dpaste_de.el'
newer than byte-compiled file
ad-handle-definition: `find-tag-noselect' got redefined
Loading term/xterm...done
6am is refreshing

Load-path shadows:
/Users/Eddie/.emacs.d/lisp/bundle/helm-20160121.2157/helm-multi-match hides
/Users/Eddie/.emacs.d/lisp/bundle/helm-core-20160121.2157/helm-multi-match

Features:
(shadow sort my-gnus mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils
xterm elisp-slime-nav help-mode etags eldoc my-conf my-lisp my-org
finder-inf which-key my-paste paste url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap dpaste_de
web s ucs-normalize rx time-stamp dash browse-url json url-parse
auth-source gnus-util time-date mm-util mail-prsvr password-cache
url-vars my-tmux emux my-helm helm helm-source helm-multi-match helm-lib
dired helm-config helm-easymenu cl-macs gv async-bytecomp async my-evil
evil-jumper evil-leader evil evil-integration undo-tree diff evil-maps
evil-commands evil-command-window evil-types evil-search evil-ex
evil-macros evil-repeat evil-states evil-core advice help-fns
evil-common windmove thingatpt rect evil-digraphs evil-vars ring
my-utils my-graphics use-package diminish bind-key easy-mmode info
easymenu eieio byte-opt bytecomp byte-compile cl-extra cconv eieio-core
package epg-config edmacro kmacro cl-loaddefs cl-lib tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel ns-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
cocoa ns multi-tty emacs)

Memory information:
((conses 16 314308 152941)
 (symbols 48 29606 97)
 (miscs 40 32 184)
 (strings 32 47915 121314)
 (string-bytes 1 1321209)
 (vectors 16 42457)
 (vector-slots 8 705376 93653)
 (floats 8 109 567)
 (intervals 56 217 74)
 (buffers 960 12))

[Message part 2 (text/html, inline)]

Message #8 received at 22744 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Shiyao Ma <i <at> introo.me>
Cc: 22744 <at> debbugs.gnu.org, wmperry <at> gnu.org
Subject: Re: bug#22744: 24.5;
 url-retrieve callback is not invoked when http response content is
 empty
Date: Sat, 20 Feb 2016 18:41:08 +1100

Shiyao Ma <i <at> introo.me> writes:

> A working snippet is here:
>
> https://bpaste.net/show/d2bd28c10f46
>
> The callback paste-callback-bpaste is not invoked.
>
> Turning on `url-debug', we can observe that the post response is a mere
> http header with status 302, and no *data content*.

A 302 is a redirection, and URL follows redirections, and do not call
the callback before we get to the "real" page (or it fails).

Is there nothing else returned from the server?  Like a Location header?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #11 received at 22744 <at> debbugs.gnu.org (full text, mbox):

From: Shiyao Ma <i <at> introo.me>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 22744 <at> debbugs.gnu.org, wmperry <at> gnu.org
Subject: Re: bug#22744: 24.5; url-retrieve callback is not invoked when http
 response content is empty
Date: Sat, 20 Feb 2016 19:15:27 +0800

[Message part 1 (text/plain, inline)]

To add more detail,

The url-debug output:
https://bpaste.net/show/2945ab6727be


A sample post response:
https://bpaste.net/show/25777c5fde69
No response content, the result is merely the http headers.


Also, url-retrieve doesn't follow the 302 redirection.

[Message part 2 (text/html, inline)]

Message #14 received at 22744 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Shiyao Ma <i <at> introo.me>
Cc: 22744 <at> debbugs.gnu.org
Subject: Re: bug#22744: 24.5;
 url-retrieve callback is not invoked when http response content is
 empty
Date: Sun, 21 Feb 2016 13:57:52 +1100

Shiyao Ma <i <at> introo.me> writes:

> A sample post response:
> https://bpaste.net/show/25777c5fde69
> No response content, the result is merely the http headers.

So you do a POST, and then you get a 302 redirect (which results in a
GET)?  I think I remember somebody working on this issue a few weeks
ago...

Oh, yeah:

commit 46dfdd831b817ef9e281350043bd4231f2dc5acc
Author: Nicolas Petton <nicolas <at> petton.fr>
Date:   Thu Feb 4 21:43:42 2016 +0100

    Do not ignore redirections of 301, 302 and 307 status codes
    
    The current version of HTTP/1.1 (RFC 7231) no longer requires
    confirmation on 301, 302 or 307 status codes, therefore we do not have
    to ignore redirects for other requests than GET and HEAD.
    
    * lisp/url/url-http.el (url-http-parse-headers): Do not ignore 301, 302
    and 307 redirects for other requests than GET and HEAD.

So this probably works in emacs-25 now.  Could you download (from git)
and test?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #17 received at 22744 <at> debbugs.gnu.org (full text, mbox):

From: Shiyao Ma <i <at> introo.me>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 22744 <at> debbugs.gnu.org
Subject: Re: bug#22744: 24.5; url-retrieve callback is not invoked when http
 response content is empty
Date: Sun, 21 Feb 2016 18:51:28 +0800

[Message part 1 (text/plain, inline)]

I installed from the git head.


For the callback status, it's:
(:error (error http 405) :redirect https://bpaste.net/show/9f7b83e91ebc)


For the callback buffer, it's:
HTTP/1.1 405 METHOD NOT ALLOWED
Server: nginx
Date: Sun, 21 Feb 2016 10:45:21 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Allow: HEAD, OPTIONS, GET
X-Cache-Hits: 0
X-Cache-Age: 0
X-Cache-Status: MISS

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>



For the url-debug output, it's:
https://bpaste.net/show/f5b3e99df8fd




*****

So from the url-debug output, the logic is doing wrong. On Line#30, it's
doing a *redirect* with post, resulting an HTTP 405.



BTW, possible to do nothing other than firing up the callback when
receiving the HTTP 302  on Line#32 ?


Regards.

[Message part 2 (text/html, inline)]

Message #20 received at 22744 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Shiyao Ma <i <at> introo.me>
Cc: Nicolas Petton <nicolas <at> petton.fr>, 22744 <at> debbugs.gnu.org
Subject: Re: bug#22744: 24.5;
 url-retrieve callback is not invoked when http response content is
 empty
Date: Mon, 22 Feb 2016 14:03:00 +1100

Shiyao Ma <i <at> introo.me> writes:

> I installed from the git head.
>
> For the callback status, it's:
> (:error (error http 405) :redirect https://bpaste.net/show/9f7b83e91ebc)
>
> For the callback buffer, it's:
> HTTP/1.1 405 METHOD NOT ALLOWED
> Server: nginx

[...]

> So from the url-debug output, the logic is doing wrong. On Line#30, it's doing
> a *redirect* with post, resulting an HTTP 405.

So that web site wants the redirect to be done with GET instead of POST?
I don't know what the standard says should happen here...

Nicolas, could you look into this?  The issue is that when POST-ing to
bpaste.net, it 302 redirects...

> BTW, possible to do nothing other than firing up the callback when receiving
> the HTTP 302 on Line#32 ?

No, the callbacks are fired after landing on the final URL after all the
redirects.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #23 received at 22744 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Shiyao Ma <i <at> introo.me>
Cc: Nicolas Petton <nicolas <at> petton.fr>, 22744 <at> debbugs.gnu.org
Subject: Re: bug#22744: 24.5; url-retrieve callback is not invoked when http
 response content is empty
Date: Tue, 24 Sep 2019 10:14:04 +0200

Lars Ingebrigtsen <larsi <at> gnus.org> writes:

>> So from the url-debug output, the logic is doing wrong. On Line#30, it's doing
>> a *redirect* with post, resulting an HTTP 405.
>
> So that web site wants the redirect to be done with GET instead of POST?
> I don't know what the standard says should happen here...

This was apparently fixed in 2018 like this:

           ('found			; 302
	    ;; 302 Found was ambiguously defined in the standards, but
	    ;; it's now recommended that it's treated like 303 instead
	    ;; of 307, since that's what most servers expect.
	    (setq url-http-method "GET"
		  url-http-data nil))

To I think it's fixed, and I'm closing this bug report.  If you're
seeing a problem here, please reopen.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.