GNU bug report logs - #61916
30.0.50; url.el; Can not change User-Agent temporarily when url-retrieve

Previous Next

To reply to this bug, email your comments to 61916 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: xhcoding <xhcoding <at> foxmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.50; url.el;
 Can not change User-Agent temporarily when url-retrieve
Date: Thu, 02 Mar 2023 21:37:13 +0800

The following code cannot be used to temporarily change User-Agent

(require 'url)
(setq url-debug t)
(let ((url-user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.57"))
  (url-retrieve "https://httpbin.org/get?a=b" (lambda (_))))


Because the url-retrieve is asynchronous, when the url-user-agent is actually used (eg. url-http-create-request be called),
it may be out of the let scope.



In GNU Emacs 30.0.50 (build 5, x86_64-w64-mingw32)
Repository revision: 07a76a06f9a438e2a1bf1e565e3e78f7927c37f2
Repository branch: master
Windowing system distributor 'Microsoft Corp.', version 10.0.22621
System Description: Microsoft Windows 10 Home China (v10.0.2009.22621.1105)

Configured using:
 'configure --prefix=/d/emacs-build/pkg/master-x86_64 'CFLAGS=-Ofast -fno-finite-math-only -fomit-frame-pointer'
 --with-zlib --with-xpm --with-xml2 --with-tree-sitter --without-tiff --with-rsvg --with-png --with-lcms2 --with-json
 --with-jpeg --with-harfbuzz --with-gnutls --with-gif --without-cairo --disable-build-details --without-dbus
 --with-compress-install --with-small-ja-dic'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: CHS
  locale-coding-system: cp936

Major mode: ELisp/d

Message #8 received at 61916 <at> debbugs.gnu.org (full text, mbox):

From: "Peder O. Klingenberg" <peder <at> klingenberg.no>
To: xhcoding <xhcoding <at> foxmail.com>
Cc: 61916 <at> debbugs.gnu.org
Subject: Re: bug#61916: 30.0.50; url.el; Can not change User-Agent
 temporarily when url-retrieve
Date: Fri, 18 Jul 2025 14:57:58 +0200

tags 61916 + patch
quit

On Thu, 2023-03-02 21:37:13 +0800, xhcoding wrote:

> The following code cannot be used to temporarily change User-Agent
>
> (require 'url)
> (setq url-debug t)
> (let ((url-user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.57"))
>   (url-retrieve "https://httpbin.org/get?a=b" (lambda (_))))

Some, but not all, variables used to construct a request by url-http.el
have their dynamic values preserved in the buffer used for the async
communication. url-user-agent was not among the so blessed.  For the
unlucky variables, the behaviour is extra confusing because connections
are sometimes re-used (due to http keep-alive), in which case the
dynamic bindings _do_ work as expected.

This same issue hit me as I was working on improving restclient.el,
where some of the bug reports probably originate from this confusing
behaviour of url-http.el.

Attached is a patch that extends the set of dynamic-values-at-call-time
-> buffer-local-available-at-request-time variables.  With this, your
test case works as expected, as do my experiments in restclient.

Hopefully this will be picked up by someone with commit privileges.

-- 
...Peder...

Message #13 received at 61916 <at> debbugs.gnu.org (full text, mbox):

From: "Peder O. Klingenberg" <peder <at> klingenberg.no>
To: xhcoding <xhcoding <at> foxmail.com>
Cc: 61916 <at> debbugs.gnu.org
Subject: Re: bug#61916: 30.0.50; url.el; Can not change User-Agent
 temporarily when url-retrieve
Date: Fri, 18 Jul 2025 15:03:49 +0200

[Message part 1 (text/plain, inline)]

On Fri, 2025-07-18 14:57:58 +0200, Peder O. Klingenberg wrote:

> Attached is a patch that extends the set of dynamic-values-at-call-time
> -> buffer-local-available-at-request-time variables.  With this, your
> test case works as expected, as do my experiments in restclient.

(Except I failed to attach the patch using the debbugs-gnu-pick-commits
command, so here, attached the old-fashioned way)

-- 
...Peder...

[0001-Emulate-more-dynamic-bindings-in-request-buffers.patch (text/x-diff, attachment)]

Message #18 received at 61916-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: "Peder O. Klingenberg" <peder <at> klingenberg.no>
Cc: xhcoding <at> foxmail.com, 61916-done <at> debbugs.gnu.org
Subject: Re: bug#61916: 30.0.50; url.el;
 Can not change User-Agent temporarily when url-retrieve
Date: Sat, 26 Jul 2025 12:11:43 +0300

> Cc: 61916 <at> debbugs.gnu.org
> From: "Peder O. Klingenberg" <peder <at> klingenberg.no>
> Date: Fri, 18 Jul 2025 15:03:49 +0200
> 
> On Fri, 2025-07-18 14:57:58 +0200, Peder O. Klingenberg wrote:
> 
> > Attached is a patch that extends the set of dynamic-values-at-call-time
> > -> buffer-local-available-at-request-time variables.  With this, your
> > test case works as expected, as do my experiments in restclient.
> 
> (Except I failed to attach the patch using the debbugs-gnu-pick-commits
> command, so here, attached the old-fashioned way)

Thanks, installed on the master branch, and closing the bug.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.