GNU bug report logs - #11981
24.1.50; url-http-parse-headers should not disable file name handlers since it breaks auth-source

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 11981 in the body.
You can then email your comments to 11981 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: David Engster <deng <at> randomsample.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Wed, 18 Jul 2012 20:38:48 +0200

The function `url-http-parse-headers' temporarily disables all file name
handlers to circumvent a bug where Tramp could kick in because the URL
might look like a remote file (#6717). This is the offending snippet:

	;; The filename part of a URL could be in remote file syntax,
	;; see Bug#6717 for an example.  We disable file name
	;; handlers, therefore.
	(file-name-handler-alist nil))

However, this workaraound assumes that everything that gets afterwards
in the `let'-section does not depend on file name handlers. However,
this is not the case: `url-http-handle-authentication' is called which
in turn calls the auth-source package. This package however supports
encrypted authinfo files which have to be automatically decrypted
through `auto-encryption-mode'. This, of course, works through file name
handlers.

Here's a backtrace from `auth-source-search' resulting from a call to
`url-retrieve-synchronously' on a password-protected URL:

  auth-source-search(:max 1 :host "www.google.com:443" :port "https")
  url-do-auth-source-search("www.google.com:443" "https" :user)
  url-basic-auth([cl-struct-url "https" nil nil "www.google.com" 443  "/calendar/dav/MY-GOOGLE-ACCOUNT/events/" nil nil t nil t] t nil "Google CalDAV" (("realm" . "Google CalDAV") ("basic")))
  url-get-authentication("https://www.google.com/calendar/dav/MY-GOOGLE-ACCOUNT/events/" "Google CalDAV" "basic" t (("realm" . "Google CalDAV") ("basic")))
  url-http-handle-authentication(nil)
  [ ... ]
  url-http-parse-headers()
  url-http-chunked-encoding-after-change-function(500 507 7)
  url-http-generic-filter(#<process www.google.com<1>> "\n0\n\n")
  [ ... ]
  url-retrieve-synchronously("https://www.google.com/calendar/dav/MY-GOOGLE-ACCOUNT/events/")


Note that after `url-http-parse-headers' the variable
`file-name-handler-alist' is set to 'nil', thus when
`auth-source-search' loads an encrypted authinfo.gpg file, it won't
automatically get decrypted.

-David

Message #8 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: David Engster <deng <at> randomsample.de>
Cc: 11981 <at> debbugs.gnu.org
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Thu, 19 Jul 2012 08:35:04 +0200

David Engster <deng <at> randomsample.de> writes:

> However, this workaraound assumes that everything that gets afterwards
> in the `let'-section does not depend on file name handlers. However,
> this is not the case: `url-http-handle-authentication' is called which
> in turn calls the auth-source package. This package however supports
> encrypted authinfo files which have to be automatically decrypted
> through `auto-encryption-mode'. This, of course, works through file name
> handlers.

Does it work, if Tramp is disabled instead?

--8<---------------cut here---------------start------------->8---
*** /usr/local/src/emacs/lisp/url/url-http.el.~109155~  2012-07-19
    08:29:53.339768264 +0200
--- /usr/local/src/emacs/lisp/url/url-http.el   2012-07-19
    08:28:26.428326961 +0200
***************
*** 510,518 ****
    ;; other status symbols: jewelry and luxury cars
    (status-symbol (cadr (assq url-http-response-status url-http-codes)))
    ;; The filename part of a URL could be in remote file syntax,
!   ;; see Bug#6717 for an example.  We disable file name
!   ;; handlers, therefore.
!   (file-name-handler-alist nil))
      (setq class (/ url-http-response-status 100))
      (url-http-debug "Parsed HTTP headers: class=%d status=%d" class url-http-response-status)
      (when (url-use-cookies url-http-target-url)
--- 510,517 ----
    ;; other status symbols: jewelry and luxury cars
    (status-symbol (cadr (assq url-http-response-status url-http-codes)))
    ;; The filename part of a URL could be in remote file syntax,
!   ;; see Bug#6717 for an example.  We disable Tramp, therefore.
!   (tramp-mode nil))
      (setq class (/ url-http-response-status 100))
      (url-http-debug "Parsed HTTP headers: class=%d status=%d" class url-http-response-status)
      (when (url-use-cookies url-http-target-url)
--8<---------------cut here---------------end--------------->8---

> -David

Best regards, Michael.

Message #11 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 11981 <at> debbugs.gnu.org, David Engster <deng <at> randomsample.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Thu, 19 Jul 2012 04:55:54 -0400

>> However, this workaraound assumes that everything that gets afterwards
>> in the `let'-section does not depend on file name handlers. However,
>> this is not the case: `url-http-handle-authentication' is called which
>> in turn calls the auth-source package. This package however supports
>> encrypted authinfo files which have to be automatically decrypted
>> through `auto-encryption-mode'. This, of course, works through file name
>> handlers.
> Does it work, if Tramp is disabled instead?

I think I'd first like to better understand bug#6717: why do we take
a local part of a URL, let it start with / and then pass it to
file-name-directory?  That sounds like a problem in itself.


        Stefan

Message #14 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: David Engster <deng <at> randomsample.de>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 11981 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Thu, 19 Jul 2012 14:51:07 +0200

Stefan Monnier writes:
>>> However, this workaraound assumes that everything that gets afterwards
>>> in the `let'-section does not depend on file name handlers. However,
>>> this is not the case: `url-http-handle-authentication' is called which
>>> in turn calls the auth-source package. This package however supports
>>> encrypted authinfo files which have to be automatically decrypted
>>> through `auto-encryption-mode'. This, of course, works through file name
>>> handlers.
>> Does it work, if Tramp is disabled instead?
>
> I think I'd first like to better understand bug#6717: why do we take
> a local part of a URL, let it start with / and then pass it to
> file-name-directory?  That sounds like a problem in itself.

I agree. I wonder which call to `file-name-directory' threw the error in
the original report - the most likely candidate is `url-file-directory',
which should simply be rewritten to not use `file-name-directory'.

-David

Message #17 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: David Engster <deng <at> randomsample.de>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 11981 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Sat, 21 Jul 2012 16:31:18 +0200

[Message part 1 (text/plain, inline)]

David Engster writes:
> Stefan Monnier writes:
>>>> However, this workaraound assumes that everything that gets afterwards
>>>> in the `let'-section does not depend on file name handlers. However,
>>>> this is not the case: `url-http-handle-authentication' is called which
>
>>>> in turn calls the auth-source package. This package however supports
>>>> encrypted authinfo files which have to be automatically decrypted
>>>> through `auto-encryption-mode'. This, of course, works through file name
>>>> handlers.
>>> Does it work, if Tramp is disabled instead?
>>
>> I think I'd first like to better understand bug#6717: why do we take
>> a local part of a URL, let it start with / and then pass it to
>> file-name-directory?  That sounds like a problem in itself.
>
> I agree. I wonder which call to `file-name-directory' threw the error in
> the original report - the most likely candidate is `url-file-directory',
> which should simply be rewritten to not use `file-name-directory'.

Patch attached. It even fixes another bug along the way:
`url-file-(non)directory' would not work on hexified URLs.

I'm not 100% sure this also fixes bug #6717; it might be that there's
another file-* call lurking somewhere.

-David

[url-util-diff.patch (text/x-patch, inline)]

=== modified file 'lisp/url/url-util.el'
--- lisp/url/url-util.el	2012-07-11 23:13:41 +0000
+++ lisp/url/url-util.el	2012-07-21 14:25:59 +0000
@@ -246,18 +246,24 @@
   "Return the directory part of FILE, for a URL."
   (cond
    ((null file) "")
-   ((string-match "\\?" file)
-    (file-name-directory (substring file 0 (match-beginning 0))))
-   (t (file-name-directory file))))
+   ((string-match "\\(\\?\\|%3[fF]\\)" file)
+    (url-file-directory (substring file 0 (match-beginning 0))))
+   ((string-match "\\(.*\\(/\\|%2[fF]\\)\\)" file)
+    (match-string 1 file))
+   (t
+    "")))
 
 ;;;###autoload
 (defun url-file-nondirectory (file)
   "Return the nondirectory part of FILE, for a URL."
   (cond
    ((null file) "")
-   ((string-match "\\?" file)
-    (file-name-nondirectory (substring file 0 (match-beginning 0))))
-   (t (file-name-nondirectory file))))
+   ((string-match "\\(\\?\\|%3[fF]\\)" file)
+    (url-file-nondirectory (substring file 0 (match-beginning 0))))
+   ((string-match ".*\\(?:/\\|%2[fF]\\)\\(.*\\)" file)
+    (match-string 1 file))
+   (t
+    "")))
 
 ;;;###autoload
 (defun url-parse-query-string (query &optional downcase allow-newlines)

Message #20 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
To: David Engster <deng <at> randomsample.de>
Cc: 11981 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Mon, 06 Aug 2012 12:01:13 -0400

> @@ -246,18 +246,24 @@
>    "Return the directory part of FILE, for a URL."
>    (cond
>     ((null file) "")
> -   ((string-match "\\?" file)
> -    (file-name-directory (substring file 0 (match-beginning 0))))
> -   (t (file-name-directory file))))
> +   ((string-match "\\(\\?\\|%3[fF]\\)" file)
> +    (url-file-directory (substring file 0 (match-beginning 0))))

Should %3F really be treated like a ? (i.e. a special char that
delimits two different parts of a URL) rather than like "a normal ?
character encoded so as not to delimit two different parts of a URL"?


        Stefan

Message #23 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: Jan Djärv <jan.h.d <at> swipnet.se>
To: Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Cc: 11981 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
	David Engster <deng <at> randomsample.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Mon, 6 Aug 2012 19:18:21 +0200

Hello.

6 aug 2012 kl. 18:01 skrev Stefan Monnier:

>> @@ -246,18 +246,24 @@
>>   "Return the directory part of FILE, for a URL."
>>   (cond
>>    ((null file) "")
>> -   ((string-match "\\?" file)
>> -    (file-name-directory (substring file 0 (match-beginning 0))))
>> -   (t (file-name-directory file))))
>> +   ((string-match "\\(\\?\\|%3[fF]\\)" file)
>> +    (url-file-directory (substring file 0 (match-beginning 0))))
> 
> Should %3F really be treated like a ? (i.e. a special char that
> delimits two different parts of a URL) rather than like "a normal ?
> character encoded so as not to delimit two different parts of a URL"?
> 

A normal non-delimiting ?.  How else can one get a ? in to an URL?

	Jan D.

Message #26 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: David Engster <deng <at> randomsample.de>
To: Jan Djärv <jan.h.d <at> swipnet.se>
Cc: 11981 <at> debbugs.gnu.org, Michael Albinus <michael.albinus <at> gmx.de>,
	Stefan Monnier <monnier <at> IRO.UMontreal.CA>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Sun, 12 Aug 2012 16:52:47 +0200

Jan Djärv writes:
> Hello.
>
> 6 aug 2012 kl. 18:01 skrev Stefan Monnier:
>
>>> @@ -246,18 +246,24 @@
>>>   "Return the directory part of FILE, for a URL."
>>>   (cond
>>>    ((null file) "")
>>> -   ((string-match "\\?" file)
>>> -    (file-name-directory (substring file 0 (match-beginning 0))))
>>> -   (t (file-name-directory file))))
>>> +   ((string-match "\\(\\?\\|%3[fF]\\)" file)
>>> +    (url-file-directory (substring file 0 (match-beginning 0))))
>> 
>> Should %3F really be treated like a ? (i.e. a special char that
>> delimits two different parts of a URL) rather than like "a normal ?
>> character encoded so as not to delimit two different parts of a URL"?
>> 
>
> A normal non-delimiting ?.  How else can one get a ? in to an URL?

You are both right, of course; I was overcompensating. The match for
%3[fF] must be removed.

-David

Message #29 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: David Engster <deng <at> randomsample.de>
Cc: 11981 <at> debbugs.gnu.org, Jan Djärv <jan.h.d <at> swipnet.se>,
	Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Sun, 12 Aug 2012 13:36:31 -0400

> You are both right, of course; I was overcompensating. The match for
> %3[fF] must be removed.

Thanks, I installed your patch with this part adjusted (as well as the
default case changed to better match file-name-(non)directory's
behavior).
Can we now re-enable file-name-handlers in url-http-parse-headers?


        Stefan

Message #32 received at 11981 <at> debbugs.gnu.org (full text, mbox):

From: David Engster <deng <at> randomsample.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 11981 <at> debbugs.gnu.org, Jan Djärv <jan.h.d <at> swipnet.se>,
	Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Mon, 13 Aug 2012 21:23:01 +0200

Stefan Monnier writes:
>> You are both right, of course; I was overcompensating. The match for
>> %3[fF] must be removed.
>
> Thanks, I installed your patch with this part adjusted (as well as the
> default case changed to better match file-name-(non)directory's
> behavior).
> Can we now re-enable file-name-handlers in url-http-parse-headers?

I think so. I re-enabled the file-name-handlers, created a test feed
with a colon in it

http://www.randomsample.de/foo:bar.xml

and I could subscribe to it with Gnus (the original URLs in Bug #6717
gave me http 405 errors).

-David

Message #37 received at 11981-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: David Engster <deng <at> randomsample.de>
Cc: 11981-done <at> debbugs.gnu.org,
	Jan Djärv <jan.h.d <at> swipnet.se>,
	Michael Albinus <michael.albinus <at> gmx.de>
Subject: Re: bug#11981: 24.1.50;
	url-http-parse-headers should not disable file name handlers since it
	breaks auth-source
Date: Tue, 14 Aug 2012 10:55:39 -0400

>> Can we now re-enable file-name-handlers in url-http-parse-headers?

> I think so. I re-enabled the file-name-handlers, created a test feed
> with a colon in it
> http://www.randomsample.de/foo:bar.xml
> and I could subscribe to it with Gnus (the original URLs in Bug #6717
> gave me http 405 errors).

OK, re-enabled, then, thank you,


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.