GNU bug report logs - #4883
23.1; url-http: wrong server name used for pwd lookup during proxy authentication

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 4883 in the body.
You can then email your comments to 4883 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Kai Tetzlaff <kai.tetzlaff <at> web.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 23.1; url-http: wrong server name used for pwd lookup during proxy authentication
Date: Sat, 07 Nov 2009 12:43:01 +0100

Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your bug report will be posted to the bug-gnu-emacs <at> gnu.org mailing list,
and to the gnu.emacs.bug news group.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

The following describes a problem with password lookup which happens when
trying to download an HTTP document through a proxy server when this proxy
server requires users to authenticate. When trying to find credentials, the
URL lib is using the name of the server in the original download URL instead
of the name of the proxy server for the username/password lookup.

I've tried to analyze what's happening and also have a patch (at the end of
this bug report). But now some more details:

Even though this report describes what i believe is a bug in the url lib, let
me first explain shortly what i've been doing when i discovered the problem:
I've been trying out the irfc package (http://www.emacswiki.org/emacs/irfc.el)
to download RFCs from emacs. At work i can only access the web via an HTTP
proxy server which requires (http basic) authentication.

Using:

(setq url-proxy-services '(("http" . "proxy.example.net:80")))

i then tried to download rfc3098 using irfc-visit. Irfc-visit subsequently
calls url-retrieve with the result of:

(url-generic-parse-url "http://www.ietf.org/rfc/rfc3098.txt")

The functions in the url.el and friends start doing their work. A GET request
gets sent to the proxy server which is answered with a 407 response asking for
authentication. The URL lib detects the authentication request and tries to
obtain the required credentials. This is done by calling:

...
url-get-authentication (url-auth.el)
url-basic-auth (url-auth.el)

and finally:
 
auth-source-user-or-password (auth-source.el)

which uses the name of the server which asks for authentication (and a
protocol identifier) to check if a login/password is stored somewhere (in my
case: ~/authinfo.gpg).

However, the lookup is using www.ietf.org to lookup the password instead of
the name of the proxy server (proxy.example.net). So the lookup fails.

The following patch tries to fix proxy authentication in
url-http-handle-authentication. It is adding a new variable auth-url which
gets set to the proxy url or the original download URL depending on the use of
a proxy server. This new variable is then used in the call to
url-get-authentication (instead of always using the original download URL).

There might be a better solution from someone who has a deeper understanding
of the code in the url lib. But the following solves the problem - at least
for my case:

Index: lisp/url/url-http.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/url/url-http.el,v
retrieving revision 1.64
diff -u -r1.64 url-http.el
--- lisp/url/url-http.el	1 Oct 2009 02:06:55 -0000	1.64
+++ lisp/url/url-http.el	6 Nov 2009 10:25:02 -0000
@@ -315,12 +315,16 @@
 		  '("basic")))
 	(type nil)
 	(url (url-recreate-url url-current-object))
-	(url-basic-auth-storage 'url-http-real-basic-auth-storage)
+	(auth-url (url-recreate-url
+		   (if (and proxy (boundp 'url-http-proxy))
+		       url-http-proxy
+		     url-current-object)))
+	(url-basic-auth-storage (if proxy
+				    ;; Cheating, but who cares? :)
+				    'url-http-proxy-basic-auth-storage
+				  'url-http-real-basic-auth-storage))
 	auth
 	(strength 0))
-    ;; Cheating, but who cares? :)
-    (if proxy
-	(setq url-basic-auth-storage 'url-http-proxy-basic-auth-storage))
 
     ;; find strongest supported auth
     (dolist (this-auth auths)
@@ -347,7 +351,7 @@
 		  " send it to " url-bug-address ".<hr>")
 	  (setq status t))
       (let* ((args (url-parse-args (subst-char-in-string ?, ?\; auth)))
-	     (auth (url-get-authentication url (cdr-safe (assoc "realm" args))
+	     (auth (url-get-authentication auth-url (cdr-safe (assoc "realm" args))
 					   type t args)))
 	(if (not auth)
 	    (setq success t)


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
If you would like to further debug the crash, please read the file
c:/Programme/emacs-23.1/etc/DEBUG for instructions.


In GNU Emacs 23.1.1 (i386-mingw-nt5.1.2600)
 of 2009-07-30 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.4)'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: DEU
  value of $XMODIFIERS: nil
  locale-coding-system: cp1252
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  icicle-mode: t
  minibuffer-depth-indicate-mode: t
  shell-dirtrack-mode: t
  icomplete-mode: t
  pretty-control-l-mode: t
  delete-selection-mode: t
  show-paren-mode: t
  global-auto-revert-mode: t
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<help-echo> <help-echo> <help-echo> <help-echo> <help-echo> 
<menu-bar> <help-menu> <send-emacs-bug-report>

Recent messages:
Turning OFF Icicle mode...done
ad-handle-definition: `choose-completion-string' got redefined
Turning ON Icicle mode...done
ad-handle-definition: `choose-completion-string' got redefined
Turning OFF Icicle mode...done
Loading c:/kt/.emacs.d/init/rc/rc-misc.el (source)...done
Loading c:/kt/.emacs.d/tmp/session.el (source)...done
ad-handle-definition: `choose-completion-string' got redefined
Turning ON Icicle mode...done
Finished .emacs

Message #10 received at 4883-done <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Chong Yidong <cyd <at> stupidchicken.com>
To: Kai Tetzlaff <kai.tetzlaff <at> web.de>
Cc: 4883-done <at> debbugs.gnu.org
Subject: Re: 23.1; url-http: wrong server name used for pwd lookup during proxy authentication
Date: Sun, 08 Nov 2009 00:33:48 -0500

> The following patch tries to fix proxy authentication in
> url-http-handle-authentication. It is adding a new variable auth-url
> which gets set to the proxy url or the original download URL depending
> on the use of a proxy server. This new variable is then used in the
> call to url-get-authentication (instead of always using the original
> download URL).

The patch looks good.  I've checked it into CVS; thanks.

Message #15 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Kai Tetzlaff <kai.tetzlaff <at> web.de>
To: Ted Zlatanov <tzz <at> lifelogs.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#4883: 23.1;   url-http: wrong server name used for pwd lookup
 during proxy   authentication
Date: Tue, 24 Nov 2009 07:36:43 +0100

Hi Ted,

I'm not sure why, but also my last post below never made it to the list.
I also sent it to you directly but did not get any reply so i assume
that it also got lost. I don't get why, but i'm trying again.

In addition to what i wrote below, i've now actually tested the url code
with a double auth. scenario (proxy auth + auth at url target). It
actually works fine without any further changes!

When you wrote your message suggesting to rework the patch, what problem
did you had in mind? Is there some other scenario which needs testing?

BR,
Kai



At Tue, 17 Nov 2009 09:19:00 -0600,
Ted Zlatanov wrote:
> >
> > The following message is a courtesy copy of an article
> > that has been posted to gnu.emacs.bug as well.
> >
> > On Tue, 10 Nov 2009 14:07:39 -0600 Ted Zlatanov <tzz <at> lifelogs.com>
wrote:
> >
> > TZ> On Sat, 07 Nov 2009 12:43:01 +0100 Kai Tetzlaff
<kai.tetzlaff <at> web.de> wrote:
> > KT> There might be a better solution from someone who has a deeper
understanding
> > KT> of the code in the url lib. But the following solves the problem
- at least
> > KT> for my case:
> > KT> Index: lisp/url/url-http.el
> > KT> ===================================================================
> > KT> RCS file: /cvsroot/emacs/emacs/lisp/url/url-http.el,v
> > KT> retrieving revision 1.64
> > KT> diff -u -r1.64 url-http.el
> > KT> --- lisp/url/url-http.el	1 Oct 2009 02:06:55 -0000	1.64
> > KT> +++ lisp/url/url-http.el	6 Nov 2009 10:25:02 -0000
> > KT> @@ -315,12 +315,16 @@
> > KT> '("basic")))
> > KT> (type nil)
> > KT> (url (url-recreate-url url-current-object))
> > KT> -	(url-basic-auth-storage 'url-http-real-basic-auth-storage)
> > KT> +	(auth-url (url-recreate-url
> > KT> +		   (if (and proxy (boundp 'url-http-proxy))
> > KT> +		       url-http-proxy
> > KT> +		     url-current-object)))
> > KT> +	(url-basic-auth-storage (if proxy
> > KT> +				    ;; Cheating, but who cares?  :)
> > KT> +				    'url-http-proxy-basic-auth-storage
> > KT> +				  'url-http-real-basic-auth-storage))
> > KT> auth
> > KT> (strength 0))
> > KT> -    ;; Cheating, but who cares?  :)
> > KT> -    (if proxy
> > KT> -	(setq url-basic-auth-storage 'url-http-proxy-basic-auth-storage))
> >
> > KT> ;; find strongest supported auth
> > KT> (dolist (this-auth auths)
> > KT> @@ -347,7 +351,7 @@
> > KT> " send it to " url-bug-address ".<hr>")
> > KT> (setq status t))
> > KT> (let* ((args (url-parse-args (subst-char-in-string ?, ?\; auth)))
> > KT> -	     (auth (url-get-authentication url (cdr-safe (assoc
"realm" args))
> > KT> +	     (auth (url-get-authentication auth-url (cdr-safe (assoc
"realm" args))
> > KT> type t args)))
> > KT> (if (not auth)
> > KT> (setq success t)
> >
> > TZ> I introduced the auth-source hooks in url-http.el but the
question of
> > TZ> which URL needs to be authenticated is not something I
considered.  Can
> > TZ> we hook the auth at a lower point?  I'm concerned that if the
proxy and
> > TZ> the destination URL both need authentication, your proposal
won't work.
> >
> > Kai, can you please follow up with your opinion?
> >
> > Thanks
> > Ted
> >

Hi Ted,

thanks for lookin into this! I already send a reply a week ago - at
least i thought so. But somehow i now cannot find any trace of it.
Something must have gone wrong...

Back to the problem - i'm not sure what exactly is supposed to happen in
the situation where proxy and final destination URL both require
authentication. But i suppose, that the HTTP requests then need to
contain two sets of credentials. And you're probably right that this
currently would not work.

As i already wrote, i've so far just scratched at the surface of the
code in the url lib. More or less just enough to create aboves patch.
What did you have in mind with "...hook the auth at a lower point"? How
should we continue from here? For a start, I could just try what the
current url code actually does.

While looking at the previous problem, i was just using irfc to retrieve
RFC documents via HTTP through the proxy at work. To set up a test
environment for double authentication i could potentially just modify
the irfc configuration to point to a server that also requires
authentication. But there are probably better ways to do this? I also
wanted to ask if you know of a server which could be used for testing,
but then thought i'll check myself shortly. And - how could it be any
different - i actually found some sites which provide authentication
test pages (e.g. http://test.webdav.org/ (not sure if its really
working) or http://koivi.com/php-http-auth/).

Given that, it should not be difficult for me to do the first step (i.e.
just check how the current code deals with double auth). If, as is
expected, it does not work i could probably use some advice on how to
continue. Can you suggest a good entry point? Would you want to spent
some time on it yourself?


Kai

Message #20 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Kai Tetzlaff <kai.tetzlaff <at> web.de>
To: Ted Zlatanov <tzz <at> lifelogs.com>
Cc: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#4883: 23.1;   url-http: wrong server name used for pwd lookup
 during proxy   authentication
Date: Fri, 27 Nov 2009 22:37:45 +0100

On 25.11.2009 15:51, Ted Zlatanov wrote:
> On Tue, 24 Nov 2009 07:36:43 +0100 Kai Tetzlaff <kai.tetzlaff <at> web.de> wrote: 
> 
> KT> I'm not sure why, but also my last post below never made it to the list.
> KT> I also sent it to you directly but did not get any reply so i assume
> KT> that it also got lost. I don't get why, but i'm trying again.
> 
> No worries, this wasn't urgent :)
> 
> KT> In addition to what i wrote below, i've now actually tested the url code
> KT> with a double auth. scenario (proxy auth + auth at url target). It
> KT> actually works fine without any further changes!
> 
> Wonderful.
> 
> KT> When you wrote your message suggesting to rework the patch, what problem
> KT> did you had in mind? Is there some other scenario which needs testing?
> 
> I thought the double auth might be a problem because the code only takes
> into account the current auth.  But it works, as you say, and I don't
> see any side effects.  So I'm in favor of comitting your patch.  I don't
> know if it's "trivial," however (meaning that it doesn't require
> copyright assignment papers).  Do you want to propose it on emacs-devel
> or should I do it?  Either way, you'll get an answer about the copyright
> papers :)

Thanks for your support and the offer to help with emacs-devel. Actually
Chong Yidong already added the patch to emacs CVS (there's a short note
about it at the very end of the 'Emacs bug Tracking System' post above).
So it seems to have been trivial enough :-).

> 
> Thanks
> Ted

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.