GNU bug report logs -
#54921
eww/url mishandles IDN over proxy
Previous Next
Reported by: Andreas Schwab <schwab <at> linux-m68k.org>
Date: Wed, 13 Apr 2022 21:43:01 UTC
Severity: normal
Fixed in version 29.1
Done: Lars Ingebrigtsen <larsi <at> gnus.org>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 54921 in the body.
You can then email your comments to 54921 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Wed, 13 Apr 2022 21:43:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Andreas Schwab <schwab <at> linux-m68k.org>:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org.
(Wed, 13 Apr 2022 21:43:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
I have configured Privoxy as the http/https proxy. When visiting a URL
with an IDN address url-retrieve uses the original domain name for the
proxy connection, not the IDNA encoded name, causing Privoxy to return a
404 page. I'm not sure if that is a bug in Privoxy or in eww. I could
not find any RFC on that topic.
(eww "https://www.pimmelgate-süd.de/") returns the Privoxy 404 page,
whereas (eww "https://xn--pimmelgate-sd-7ob.de/") returns the actual web
page.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 01:22:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> I have configured Privoxy as the http/https proxy. When visiting a URL
> with an IDN address url-retrieve uses the original domain name for the
> proxy connection, not the IDNA encoded name, causing Privoxy to return a
> 404 page. I'm not sure if that is a bug in Privoxy or in eww. I could
> not find any RFC on that topic.
What does your proxy setup in Emacs look like? Is this via url-proxy
and socks.el?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 07:12:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 14 2022, Lars Ingebrigtsen wrote:
> What does your proxy setup in Emacs look like?
I have set https_proxy=http://localhost:8118/ in the environment, which
is picked up by url to set url-proxy-services.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 13:16:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> I have set https_proxy=http://localhost:8118/ in the environment, which
> is picked up by url to set url-proxy-services.
Hm, right...
I'm not at all familiar with this code, so here's a stab in the dark.
Does this fix the problem?
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index daeba17031..22e003d22d 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1405,10 +1405,12 @@ url-https-proxy-connect
(and proxy-auth
(concat "Proxy-Authorization: " proxy-auth "\r\n")))
"\r\n")
- (url-host url-current-object)
+ (url-http--encode-string
+ (puny-encode-domain (url-host url-current-object)))
(or (url-port url-current-object)
url-https-default-port)
- (url-host url-current-object))))
+ (url-http--encode-string
+ (puny-encode-domain (url-host url-current-object))))))
(defun url-https-proxy-after-change-function (_st _nd _length)
(let* ((process-buffer (current-buffer))
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 13:54:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 14 2022, Lars Ingebrigtsen wrote:
> Andreas Schwab <schwab <at> linux-m68k.org> writes:
>
>> I have set https_proxy=http://localhost:8118/ in the environment, which
>> is picked up by url to set url-proxy-services.
>
> Hm, right...
>
> I'm not at all familiar with this code, so here's a stab in the dark.
> Does this fix the problem?
>
> diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
> index daeba17031..22e003d22d 100644
> --- a/lisp/url/url-http.el
> +++ b/lisp/url/url-http.el
> @@ -1405,10 +1405,12 @@ url-https-proxy-connect
> (and proxy-auth
> (concat "Proxy-Authorization: " proxy-auth "\r\n")))
> "\r\n")
> - (url-host url-current-object)
> + (url-http--encode-string
> + (puny-encode-domain (url-host url-current-object)))
> (or (url-port url-current-object)
> url-https-default-port)
> - (url-host url-current-object))))
> + (url-http--encode-string
> + (puny-encode-domain (url-host url-current-object))))))
url-http--encode-string shouldn't be needed, since puny encoding is
always ASCII.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 13:56:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> url-http--encode-string shouldn't be needed, since puny encoding is
> always ASCII.
Just cargo-culting the other calls to punycode in URL.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 14:15:01 GMT)
Full text and
rfc822 format available.
Message #23 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 14 2022, Lars Ingebrigtsen wrote:
> Andreas Schwab <schwab <at> linux-m68k.org> writes:
>
>> url-http--encode-string shouldn't be needed, since puny encoding is
>> always ASCII.
>
> Just cargo-culting the other calls to punycode in URL.
Unlike url-http-create-request, url-https-proxy-connect doesn't have
that weird multibyte test, and can handle any ASCII string.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 14:16:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> Unlike url-http-create-request, url-https-proxy-connect doesn't have
> that weird multibyte test, and can handle any ASCII string.
OK. But did the patch fix the problem you're seeing?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 15:50:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 54921 <at> debbugs.gnu.org (full text, mbox):
error in process filter: url-https-proxy-after-change-function: Error: (error Non-ASCII hostname www.pimmelgate-süd.de detected, please use puny-encode-domain)
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 15:55:02 GMT)
Full text and
rfc822 format available.
Message #32 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> error in process filter: url-https-proxy-after-change-function: Error:
> (error Non-ASCII hostname www.pimmelgate-süd.de detected, please use
> puny-encode-domain)
Try:
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index daeba17031..06e7dbf756 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -1405,10 +1405,12 @@ url-https-proxy-connect
(and proxy-auth
(concat "Proxy-Authorization: " proxy-auth "\r\n")))
"\r\n")
- (url-host url-current-object)
+ (url-http--encode-string
+ (puny-encode-domain (url-host url-current-object)))
(or (url-port url-current-object)
url-https-default-port)
- (url-host url-current-object))))
+ (url-http--encode-string
+ (puny-encode-domain (url-host url-current-object))))))
(defun url-https-proxy-after-change-function (_st _nd _length)
(let* ((process-buffer (current-buffer))
@@ -1430,12 +1432,12 @@ url-https-proxy-after-change-function
(condition-case e
(let ((tls-connection (gnutls-negotiate
:process proc
- :hostname (url-host url-current-object)
+ :hostname (puny-encode-domain (url-host url-current-object))
:verify-error nil)))
;; check certificate validity
(setq tls-connection
(nsm-verify-connection tls-connection
- (url-host url-current-object)
+ (puny-encode-domain (url-host url-current-object))
(url-port url-current-object)))
(with-current-buffer process-buffer (erase-buffer))
(set-process-buffer tls-connection process-buffer)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 16:04:02 GMT)
Full text and
rfc822 format available.
Message #35 received at 54921 <at> debbugs.gnu.org (full text, mbox):
That works, but still fails with http://www.pimmelgate-süd.de/.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 16:12:02 GMT)
Full text and
rfc822 format available.
Message #38 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> That works, but still fails with http://www.pimmelgate-süd.de/.
OK, I've pushed the https proxy fix to master, then.
Where does the plain-HTTP proxy stuff live, then? Hm... It could be
here, I guess. Can you try the following patch?
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 96a4742956..678ee05ae5 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -195,7 +195,8 @@ url-http-find-free-connection
(unwind-protect
(let ((proc (url-open-stream host buf
(if url-using-proxy
- (url-host url-using-proxy)
+ (puny-encode-domain
+ (url-host url-using-proxy))
host)
(if url-using-proxy
(url-port url-using-proxy)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 16:46:02 GMT)
Full text and
rfc822 format available.
Message #41 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 14 2022, Lars Ingebrigtsen wrote:
> + (puny-encode-domain
> + (url-host url-using-proxy))
That's the address of the proxy, not the original URL.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 17:15:02 GMT)
Full text and
rfc822 format available.
Message #44 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
>> + (puny-encode-domain
>> + (url-host url-using-proxy))
>
> That's the address of the proxy, not the original URL.
So it is. Then I have no idea where the Host in the http case is coming
from. I'd have thought that the proxy would just look at the Host
header (which should be puny-encoded), but I don't know much about http
proxying.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Thu, 14 Apr 2022 17:40:02 GMT)
Full text and
rfc822 format available.
Message #47 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 14 2022, Lars Ingebrigtsen wrote:
> So it is. Then I have no idea where the Host in the http case is coming
> from. I'd have thought that the proxy would just look at the Host
> header (which should be puny-encoded), but I don't know much about http
> proxying.
The proxy just receives the full URL in the GET request. Something like
this, but puny-encode-domain must only be called with the domain name.
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index daeba17031..c59613da62 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -332,7 +332,8 @@ url-http-create-request
(if (and using-proxy
;; Bug#35969.
(not (equal "https" (url-type url-http-target-url))))
- (url-recreate-url url-http-target-url) real-fname))
+ (puny-encode-domain (url-recreate-url url-http-target-url))
+ real-fname))
" HTTP/" url-http-version "\r\n"
;; Version of MIME we speak
"MIME-Version: 1.0\r\n"
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Fri, 15 Apr 2022 08:36:01 GMT)
Full text and
rfc822 format available.
Message #50 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
> The proxy just receives the full URL in the GET request. Something like
> this, but puny-encode-domain must only be called with the domain name.
Right. Does the following untested patch work?
diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el
index 96a4742956..b5bcd123c7 100644
--- a/lisp/url/url-http.el
+++ b/lisp/url/url-http.el
@@ -332,7 +332,10 @@ url-http-create-request
(if (and using-proxy
;; Bug#35969.
(not (equal "https" (url-type url-http-target-url))))
- (url-recreate-url url-http-target-url) real-fname))
+ (let ((url (copy-sequence url-http-target-url)))
+ (setf (url-host url) (puny-encode-domain (url-host url)))
+ (url-recreate-url url))
+ real-fname))
" HTTP/" url-http-version "\r\n"
;; Version of MIME we speak
"MIME-Version: 1.0\r\n"
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Fri, 15 Apr 2022 11:26:02 GMT)
Full text and
rfc822 format available.
Message #53 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 15 2022, Lars Ingebrigtsen wrote:
> Right. Does the following untested patch work?
I think it would be better to expand the url struct to also store the
IDNA host name.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Fri, 15 Apr 2022 11:39:01 GMT)
Full text and
rfc822 format available.
Message #56 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
>> Right. Does the following untested patch work?
>
> I think it would be better to expand the url struct to also store the
> IDNA host name.
And then have separate functions to reconstruct the URL with the IDNA or
the original domain name, depending on what we want? I don't think that
makes that much sense.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Fri, 15 Apr 2022 11:47:01 GMT)
Full text and
rfc822 format available.
Message #59 received at 54921 <at> debbugs.gnu.org (full text, mbox):
On Apr 15 2022, Lars Ingebrigtsen wrote:
> And then have separate functions to reconstruct the URL with the IDNA or
> the original domain name, depending on what we want? I don't think that
> makes that much sense.
Just add an optional argument.
--
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1
"And now for something completely different."
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#54921; Package
emacs.
(Sat, 16 Apr 2022 15:04:02 GMT)
Full text and
rfc822 format available.
Message #62 received at 54921 <at> debbugs.gnu.org (full text, mbox):
Andreas Schwab <schwab <at> linux-m68k.org> writes:
>> And then have separate functions to reconstruct the URL with the IDNA or
>> the original domain name, depending on what we want? I don't think that
>> makes that much sense.
>
> Just add an optional argument.
I don't think that is much of an improvement -- url-recreate-url is
pretty well-defined as it is.
So I've pushed the proposed patch, which I assume fixed the problem.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
bug marked as fixed in version 29.1, send any further explanations to
54921 <at> debbugs.gnu.org and Andreas Schwab <schwab <at> linux-m68k.org>
Request was from
Lars Ingebrigtsen <larsi <at> gnus.org>
to
control <at> debbugs.gnu.org.
(Sat, 16 Apr 2022 15:04:02 GMT)
Full text and
rfc822 format available.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Sun, 15 May 2022 11:24:04 GMT)
Full text and
rfc822 format available.
This bug report was last modified 3 years and 32 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.