GNU bug report logs -
#53941
27.2; socks + tor dont work with https
Previous Next
To reply to this bug, email your comments to 53941 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Fri, 11 Feb 2022 14:32:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Jacobo <gnuhacker <at> member.fsf.org>:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org.
(Fri, 11 Feb 2022 14:32:01 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Emacs can not resolve domains when it is https if you are using a socks
proxy (socks.el)
Emacs Config:
#+begin_src elisp
(setq socks-override-functions t)
(setq url-gateway-method 'socks) ; same problem without this line in conf
(require 'socks)
(setq socks-noproxy '("localhost"))
(setq socks-server
'("TOR"
"localhost"
9050
5))
#+end_src
Tor is the socks proxy
Im running tor.
When I try:
M-x eww RET gnu.org RET
It works, load http://gnu.org (HTTP in plain)
Also work with .onion domains in HTTP plain
No problems with HTTP but
When I try:
M-x eww RET https://gnu.org RET
Return an error:
Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.
In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0)
Windowing system distributor 'The X.Org Foundation', version 11.0.12101002
System Description: GNU Guix System
Configured using:
'configure
CONFIG_SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash
SHELL=/gnu/store/4y5m9lb8k3qkb1y9m02sw9w9a6hacd16-bash-minimal-5.1.8/bin/bash
--prefix=/gnu/store/c4bb68f53mw3sjf6xbyr7ba83csgjdkk-emacs-27.2
--enable-fast-install --with-modules --with-cairo
--disable-build-details'
Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS JSON PDUMPER GMP
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 14 Feb 2022 12:38:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Jacobo,
Jacobo <gnuhacker <at> member.fsf.org> writes:
> Emacs can not resolve domains when it is https if you are using a
> socks proxy (socks.el) [...] It works, load http://gnu.org (HTTP in
> plain) Also work with .onion domains in HTTP plain No problems with
> HTTP but When I try: M-x eww RET https://gnu.org RET
>
> Return an error: Bad Request
It's certainly possible (see attached). But can it be done responsibly?
In this day and age, when processes and services resolve host names in
all manner of ways, how can we be confident there won't be any leaks? At
present, the main interfaces to various protocol stacks (for example,
url-gw.el and friends) don't seem geared toward making those kinds of
assurances. (Not that they ought to be.)
That said, providing the building blocks on the SOCKS side doesn't seem
like the crime of the century. I've been sitting on what became the
basis for these patches for a while now, but these here were hastily
adapted and might come with some warts. Still, I believe them
straightforward enough to illustrate a basic means of achieving what
you're after.
> In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30,
I also have some examples with shims for 27 running periodically in CI.
These include a demo of using ERC to connect to Libera.Chat via SOCKS
over TLS. (But that requires an IRCv3 library, which is still a work in
progress.) If you're interested in experimenting with any of this stuff,
please let me know. That goes for anyone else out there as well. Thanks.
[0001-Set-coding-system-for-SOCKS-connections-to-binary.patch (text/x-patch, attachment)]
[0002-Add-support-for-SOCKS-4a.patch (text/x-patch, attachment)]
[0003-Support-SOCKS-RESOLVE-extension.patch (text/x-patch, attachment)]
[0004-POC-Demo-SOCKS-RESOLVE-over-HTTPS.patch (text/x-patch, attachment)]
Added tag(s) patch.
Request was from
"J.P." <jp <at> neverwas.me>
to
control <at> debbugs.gnu.org.
(Tue, 15 Feb 2022 00:32:01 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Sat, 19 Feb 2022 21:05:02 GMT)
Full text and
rfc822 format available.
Message #13 received at 53941 <at> debbugs.gnu.org (full text, mbox):
"J.P." <jp <at> neverwas.me> writes:
> Jacobo <gnuhacker <at> member.fsf.org> writes:
>> Emacs can not resolve domains when it is https if you are using a
>> socks proxy (socks.el) [...] It works, load http://gnu.org (HTTP in
>> plain) Also work with .onion domains in HTTP plain No problems with
>> HTTP but When I try: M-x eww RET https://gnu.org RET
>> Return an error: Bad Request
> It's certainly possible (see attached). But can it be done responsibly?
> In this day and age, when processes and services resolve host names in
> all manner of ways, how can we be confident there won't be any leaks?
oh, nevermind
> At present, the main interfaces to various protocol stacks (for
> example, url-gw.el and friends) don't seem geared toward making those
> kinds of assurances. (Not that they ought to be.) That said,
> providing the building blocks on the SOCKS side doesn't seem like the
> crime of the century. I've been sitting on what became the basis for
> these patches for a while now, but these here were hastily adapted and
> might come with some warts. Still, I believe them straightforward
> enough to illustrate a basic means of achieving what you're after.
>> In GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30,
> I also have some examples with shims for 27 running periodically in CI.
> These include a demo of using ERC to connect to Libera.Chat via SOCKS
> over TLS. (But that requires an IRCv3 library, which is still a work in
> progress.)
Now Ive tryed same in other computer with Trisquel 10, the problem still
happend
> If you're interested in experimenting with any of this stuff, please
> let me know. That goes for anyone else out there as well. Thanks.
yes, how can I help?
--
Emacs Lover.
FSF Member.
Free/Libre Software supporter.
stallmansupport.org - Disinformation succeeds because so many people
care deeply about injustice but do not take the time to check the facts.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 21 Feb 2022 15:03:01 GMT)
Full text and
rfc822 format available.
Message #16 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Jacobo <gnuhacker <at> member.fsf.org> writes:
>> In this day and age, when processes and services resolve host names in
>> all manner of ways, how can we be confident there won't be any leaks?
>
> oh, nevermind
I certainly don't want to discourage anyone from trying to solve this.
But DNS leaks aside, predicting what ought to be proxied still seems
like a serious undertaking (at least from my peabrained perspective).
For example, if you connect to an IRC network over Tor and click a
hyperlink in a channel, should the resulting connection also happen over
Tor? What about when the SOCKS service isn't Tor but something else,
like SSH? Should similar follow-on connections also originate from the
proxy host (your VPS or shell provider, for example)?
>> I also have some examples with shims for 27 running periodically in CI.
>> These include a demo of using ERC to connect to Libera.Chat via SOCKS
>> over TLS. (But that requires an IRCv3 library, which is still a work in
>> progress.)
>
> Now Ive tryed same in other computer with Trisquel 10, the problem still
> happend
Sorry, are you saying you repeated the steps in your original post and
got the same result (failure) on another computer? If so, that's to be
expected because Tor over SOCKS with TLS isn't supported OOTB with any
Emacs, not even 29. Apologies if I implied otherwise.
>> If you're interested in experimenting with any of this stuff, please
>> let me know. That goes for anyone else out there as well. Thanks.
>
> yes, how can I help?
When 28 comes out, you can try applying those patches. Or, if you're not
cool with that, I can give you a replacement socks.el to shadow the
original. To check whether it's working, do
M-: (boundp 'socks--errors-4) RET
or similar and then try mimicking the recipe in that last patch (the one
named POC demo something). Once that works, try adapting it to your
needs for whatever protocol (except for ERC, for which you'll need to
install an unofficial WIP version).
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 01 Mar 2022 14:31:02 GMT)
Full text and
rfc822 format available.
Message #19 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
v2. Minor corrections (another bug in existing test, etc.).
[0000-NOT-A-PATCH-v1-v2.diff (text/x-patch, attachment)]
[0001-Simplify-network-stream-opener-in-socks.el.patch (text/x-patch, attachment)]
[0002-Fix-string-encoding-bug-in-socks-tests.patch (text/x-patch, attachment)]
[0003-Add-support-for-SOCKS-4a.patch (text/x-patch, attachment)]
[0004-Support-SOCKS-RESOLVE-extension.patch (text/x-patch, attachment)]
[0005-POC-Demo-SOCKS-RESOLVE-over-HTTPS.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Wed, 02 Mar 2022 02:38:02 GMT)
Full text and
rfc822 format available.
Message #22 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
v3. Passing around an opener function was clunky, so I've opted for
passing around contact params instead. I've also gone back to explicitly
setting the coding to binary because folks may not be using
`url-open-stream' (which does this indirectly by let-binding
`coding-system-for-{read,write}').
[0000-NOT-A-PATCH-v2-v3.diff (text/x-patch, attachment)]
[0001-Simplify-network-stream-opener-in-socks.el.patch (text/x-patch, attachment)]
[0002-Fix-string-encoding-bug-in-socks-tests.patch (text/x-patch, attachment)]
[0003-Add-support-for-SOCKS-4a.patch (text/x-patch, attachment)]
[0004-Support-SOCKS-resolve-extension.patch (text/x-patch, attachment)]
[0005-POC-Demo-SOCKS-resolve-with-HTTPS.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Sun, 06 Mar 2022 02:41:02 GMT)
Full text and
rfc822 format available.
Message #25 received at 53941 <at> debbugs.gnu.org (full text, mbox):
"J.P." <jp <at> neverwas.me> writes:
> v3. Passing around an opener function was clunky, so I've opted for
> passing around contact params instead. I've also gone back to explicitly
> setting the coding to binary because folks may not be using
> `url-open-stream' (which does this indirectly by let-binding
> `coding-system-for-{read,write}').
Emacs 28.0.91 compiled with this patches, dont work, connections dont
use the proxy
--
Emacs Lover.
FSF Member.
Free/Libre Software supporter.
stallmansupport.org - Disinformation succeeds because so many people
care deeply about injustice but do not take the time to check the facts.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Sun, 06 Mar 2022 03:00:01 GMT)
Full text and
rfc822 format available.
Message #28 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Jacobo <gnuhacker <at> member.fsf.org> writes:
> "J.P." <jp <at> neverwas.me> writes:
>
>> v3. Passing around an opener function was clunky, so I've opted for
>> passing around contact params instead. I've also gone back to explicitly
>> setting the coding to binary because folks may not be using
>> `url-open-stream' (which does this indirectly by let-binding
>> `coding-system-for-{read,write}').
>
> Emacs 28.0.91 compiled with this patches, dont work, connections dont
> use the proxy
As I tried to explain up thread, the patches only get you half way
there. But perhaps that wasn't clear. You still need to do something
like the following, which is what I MemoServ'd you about (but I guess
you didn't get it). Quoting from there:
;; This works with eww. Try https://check.torproject.org
(require 'socks)
(require 'gnutls)
(require 'nsm)
(setq socks-server '("tor" "127.0.0.1" 9050 5)
socks-username "user"
socks-password ""
url-gateway-method 'socks)
(defun my-socks-open-https (orig name buffer host service &rest params)
(let ((proc (apply orig name buffer host service params)))
(advice-add 'network-lookup-address-info :override #'socks-tor-resolve)
(unwind-protect
(when (eq service 443)
(gnutls-negotiate :process proc :hostname host)
(unless (string-suffix-p ".onion" host)
(nsm-verify-connection proc host service)))
(advice-remove 'network-lookup-address-info #'socks-tor-resolve))
proc))
(defun my-url-open-stream (args)
(pcase-let ((`(,name ,buffer ,host ,service ,gateway-method) args))
(when (and (eq url-gateway-method 'socks)
(eq gateway-method 'tls))
(setq gateway-method nil))
(list name buffer host service gateway-method)))
(advice-add 'socks-open-network-stream :around #'my-socks-open-https)
(advice-add 'url-open-stream :filter-args #'my-url-open-stream)
The above is an example of what I was getting at in my initial reply
about mimicking the recipe in the last patch (the ERT test). If you have
questions about how to use it, I can help you in real time on Libera, as
we did with applying the patches. Also, please try this with emacs -Q.
Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 07 Mar 2022 07:11:02 GMT)
Full text and
rfc822 format available.
Message #31 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
v4. Include a minimal (hacky[1]) url-gw integration.
I'm now slightly of the opinion that offering no interface whatsoever is
probably too stark an approach. Ignoring the three Tor-related patches
for now, it seems that even without proper url.el integration[2], we can
still try to ensure that for most use cases, no unnecessary hackery need
apply.
Another issue is whether to address the questionable top-level advising
going on with `open-network-streams', since we're already refactoring
all the functions it affects. Assuming users exist who still have
`socks-override-functions' non-nil at load time, would it make sense to
warn them more fervently than would be done for a normal deprecation?
The thinking is that folks may be relying on this for things like
bypassing firewalls at work (and could therefore get dinged more than
usual just for upgrading Emacs).
As a start, I figured we could try and determine exactly why this
(perhaps somewhat ill-considered) top-level advising was ever instituted
in the first place[3]. AFAICT, it was mainly intended to
1. allow libraries calling `open-network-stream' (and unaware of
`socks-open-network-stream') to proxy transparently[4]
2. guard the tunneled protocol from being accidentally subject to a
recommencing of the SOCKS dialog
If anyone has better ideas, please share. Thanks.
Notes
~~~~~
[1] The second patch is new and a bit of an ugly hack. It has to do with
this change from a while back:
Do not set `url-gateway-method' in `url-https'
commit 98c58df832975b01287ef749dd5235199d4cd431
Sun Sep 28 20:00:54 2014 +0200
which made it impossible for `url-gateway-method' to be respected by
`url-open-stream' when called by `url-https'. But rather than
undoing the offending portions out of hand, it might be nicer to
first figure out how url-proxy.el is supposed to work and maybe get
it and `url-retrieve-internal' (and `url-https') more in sync and
sensitive to `url-gateway-method'.
[2] If we do end up with a proper url.el solution, it might then make
more sense to emphasize the fact that `socks-open-network-stream' is
really mostly about catering to url-gw (which it is). If that's
agreeable, we could rename the following like so:
socks-open-network-stream -> socks-url-open
socks-open-network-stream-legacy -> socks-open-network-stream
socks-open-network-stream-function -> socks-url-open-function
[3] A summary of the advice-based behavior triggered by
`socks-override-functions', assuming `socks-find-route' returns
non-nil:
| topmost function invoked | o-n-s advised | s-o-f | proxied |
|---------------------------+---------------+-------+---------|
| socks-open-network-stream | nil | t | yes |
| socks-open-network-stream | nil | nil | yes |
| socks-open-network-stream | t | t | yes |
| socks-open-network-stream | t | nil | yes |
| open-network-stream | nil | t | no |
| open-network-stream | nil | nil | no |
| open-network-stream | t | nil | no |
| open-network-stream | t | t | yes |
o-n-s: open-network-stream
s-o-f: socks-override-functions
[4] It could be argued that the 2014 commit in [1] converted gw into one
such library insofar as `url-https' is concerned.
[0000-NOT-A-PATCH-v3-v4.diff (text/x-patch, attachment)]
[0001-Simplify-network-stream-opener-in-socks.el.patch (text/x-patch, attachment)]
[0002-lisp-url-url-gw.el-url-open-stream-Honor-socks-gatew.patch (text/x-patch, attachment)]
[0003-Fix-string-encoding-bug-in-socks-tests.patch (text/x-patch, attachment)]
[0004-Add-support-for-SOCKS-4a.patch (text/x-patch, attachment)]
[0005-Support-SOCKS-resolve-extension.patch (text/x-patch, attachment)]
[0006-POC-Demo-SOCKS-resolve-with-HTTPS.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Thu, 10 Mar 2022 09:00:02 GMT)
Full text and
rfc822 format available.
Message #34 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
v5. Forgot to account for look-up failures (shocking not shocking).
Also removed hard-coded port numbers from tests.
The EWW example from earlier needs some adapting:
;; M-x eww RET https://check.torproject.org RET
(require 'socks)
(require 'gnutls)
(require 'nsm)
(defun my-socks-open-https (name buffer host service &rest params)
(let ((proc (apply #'socks-open-network-stream-legacy
name buffer host service params)))
(advice-add 'network-lookup-address-info :override #'socks-tor-resolve)
(unwind-protect
(when (eq service 443)
(gnutls-negotiate :process proc :hostname host)
(unless (string-suffix-p ".onion" host)
(nsm-verify-connection proc host service)))
(advice-remove 'network-lookup-address-info #'socks-tor-resolve))
proc))
(setq socks-server '("tor" "127.0.0.1" 9050 5)
socks-username ""
socks-password ""
url-gateway-method 'socks
socks-open-network-stream-function #'my-socks-open-https)
Let me know if you need help. Thanks.
[0000-NOT-A-PATCH-v4-v5.diff (text/x-patch, attachment)]
[0001-Simplify-network-stream-opener-in-socks.el.patch (text/x-patch, attachment)]
[0002-lisp-url-url-gw.el-url-open-stream-Honor-socks-gatew.patch (text/x-patch, attachment)]
[0003-Fix-string-encoding-bug-in-socks-tests.patch (text/x-patch, attachment)]
[0004-Add-support-for-SOCKS-4a.patch (text/x-patch, attachment)]
[0005-Support-SOCKS-resolve-extension.patch (text/x-patch, attachment)]
[0006-POC-Demo-SOCKS-resolve-with-HTTPS.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 28 Nov 2022 15:31:01 GMT)
Full text and
rfc822 format available.
Message #37 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi people, maintainers,
I've lifted some fixes and minor enhancements from my POC stuff posted
to this thread earlier this year. Nothing is directly Tor related, so I
can create a new bug report, if necessary.
The second patch fixes a problem involving SOCKS 5 error handling. It
also adds support for SOCKS 4a, which allows tools that don't speak
SOCKS 5, like socat, to resolve host names. The third addresses a couple
FIXMEs but no bugs, strictly speaking. The fourth is just a demo [1].
Happy to explain whatever in detail.
Thanks,
J.P.
[1] The fourth patch demos a possible approach for tightening the
integration between socks and url-proxy, but it's not fit for
inclusion in Emacs 29. To try it out with Tor, do something like
(setq url-proxy-services '(("https" . "socks5h://127.0.0.1:9050"))
socks-username "foo"
socks-password "")
followed by an M-x eww RET https://check.torproject.org RET. (Note
that this still leaks DNS.)
[0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch (text/x-patch, attachment)]
[0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch (text/x-patch, attachment)]
[0003-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0004-POC-30.0.50-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 28 Nov 2022 17:12:02 GMT)
Full text and
rfc822 format available.
Message #40 received at 53941 <at> debbugs.gnu.org (full text, mbox):
> From: "J.P." <jp <at> neverwas.me>
> Cc: Jacobo <gnuhacker <at> member.fsf.org>, Lars Ingebrigtsen <larsi <at> gnus.org>,
> Eli Zaretskii <eliz <at> gnu.org>
> Date: Mon, 28 Nov 2022 07:30:16 -0800
>
> I've lifted some fixes and minor enhancements from my POC stuff posted
> to this thread earlier this year. Nothing is directly Tor related, so I
> can create a new bug report, if necessary.
>
> The second patch fixes a problem involving SOCKS 5 error handling. It
> also adds support for SOCKS 4a, which allows tools that don't speak
> SOCKS 5, like socat, to resolve host names. The third addresses a couple
> FIXMEs but no bugs, strictly speaking. The fourth is just a demo [1].
> Happy to explain whatever in detail.
I'm really uncomfortable with installing these changes before the release
branch is cut. The changes are hardly trivial, some controversial even to
my eyes, even though I'm no expert on network connections. For example:
> +(defun socks-open-connection (server-info &rest stream-params)
> + "Create and initialize a SOCKS process.
> +Perform authentication if needed. Expect SERVER-INFO to take the
> +form of `socks-server' and STREAM-PARAMS to be keyword params
> +accepted by `open-network-stream'."
> (interactive)
> + (unless (plist-member stream-params :coding)
> + (setf (plist-get stream-params :coding) '(binary . binary)))
AFAIU, this constitutes an incompatible change in behavior: the default for
:coding is was never 'binary' before, it was determined from the locale's
preferences. Why are we making this change here?
> @@ -446,7 +474,11 @@ socks-send-command
> nil ; Sweet sweet success!
> (delete-process proc)
> (error "SOCKS: %s"
> - (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
> + (let ((no (or (process-get proc 'socks-reply) 99)))
> + (or (if (eq version 5) ; 99 - 90 >= length(errors)
> + (nth no socks-errors)
> + (nth (- no 90) socks--errors-4))
> + "Unknown error"))))
I don't really understand the semantics here (so maybe comments need to be
upgraded), but the old and the new versions don't look to me like equivalent
code -- why the change?
> -(when socks-override-functions
> - (advice-add 'open-network-stream :around #'socks--open-network-stream))
> +;; Libraries typically offer a "stream opener" option, such as ERC's
> +;; `erc-server-connect-function'. These provide a level of
> +;; flexibility tantamount to what this variable formerly offered.
> +(make-obsolete-variable
> + 'socks-override-functions
> + "see `socks-open-network-stream' and `socks-connect-function'." "29.1")
Why this last-minute obsolescence?
> +(defun socks-open-network-stream (name buffer host service &rest params)
> + "Open and return a connection, possibly proxied over SOCKS.
The changes in this public function are so significant that I don't
understand how they can be suggested so close to the branching.
If it is possible to add support for SOCKS 4a without affecting any
previously supported versions, I'm fine. Adding tests is also fine. But
for the rest, I think you should wait until after the release branch is cut
and install this on the master branch. Sorry, it really is too late for
such changes.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 29 Nov 2022 14:25:01 GMT)
Full text and
rfc822 format available.
Message #43 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Eli Zaretskii <eliz <at> gnu.org> writes:
> I'm really uncomfortable with installing these changes before the release
> branch is cut. The changes are hardly trivial, some controversial even to
> my eyes, even though I'm no expert on network connections.
Well, I myself am just about the furthest thing from (an expert), which
certainly doesn't comport well with dropping rash changes at the
eleventh hour. (That was rather disrespectful on my part, so shame on
me.) As such, if it's easier to revisit this once things settle down,
just ignore this email and I'll re-ping you sometime down the road.
> For example:
>
>> +(defun socks-open-connection (server-info &rest stream-params)
>> + "Create and initialize a SOCKS process.
>> +Perform authentication if needed. Expect SERVER-INFO to take the
>> +form of `socks-server' and STREAM-PARAMS to be keyword params
>> +accepted by `open-network-stream'."
>> (interactive)
>> + (unless (plist-member stream-params :coding)
>> + (setf (plist-get stream-params :coding) '(binary . binary)))
>
> AFAIU, this constitutes an incompatible change in behavior: the default for
> :coding is was never 'binary' before, it was determined from the locale's
> preferences. Why are we making this change here?
Just good old fashioned stupidity, I'm afraid. (And also recklessness in
overly trusting the me from eight months ago, surely.) I guess I somehow
assumed that if the caller didn't set :coding explicitly, they would do
so once handed back the process, which is certifiably dumb.
>> @@ -446,7 +474,11 @@ socks-send-command
>> nil ; Sweet sweet success!
>> (delete-process proc)
>> (error "SOCKS: %s"
>> - (nth (or (process-get proc 'socks-reply) 1) socks-errors)))
>> + (let ((no (or (process-get proc 'socks-reply) 99)))
>> + (or (if (eq version 5) ; 99 - 90 >= length(errors)
>> + (nth no socks-errors)
>> + (nth (- no 90) socks--errors-4))
>> + "Unknown error"))))
>
> I don't really understand the semantics here (so maybe comments need to be
> upgraded), but the old and the new versions don't look to me like equivalent
> code -- why the change?
This sets the fallback message to "Unknown error" (made up) rather than
"General SOCKS server failure" (an official error code). At first, I
figured the distinction more faithfully conveyed the nature of the
error, but now I see that it just adds clutter because the fallback path
can only be triggered by a protocol mishap, and that's unlikely, given
that the conversation must progress to its third back-and-forth by the
time this runs.
(BTW, the words "error handling" in the patch's title refer to the added
"(0 0)" `pcase' condition in `socks-filter' and not the snippet above.)
>> -(when socks-override-functions
>> - (advice-add 'open-network-stream :around #'socks--open-network-stream))
>> +;; Libraries typically offer a "stream opener" option, such as ERC's
>> +;; `erc-server-connect-function'. These provide a level of
>> +;; flexibility tantamount to what this variable formerly offered.
>> +(make-obsolete-variable
>> + 'socks-override-functions
>> + "see `socks-open-network-stream' and `socks-connect-function'." "29.1")
>
> Why this last-minute obsolescence?
Just my being callous. I now see that obsoleting that variable is
problematic, not least because we continue to honor `socks-noproxy'. But
the two complement each other and are closely coupled, usage-wise.
Getting rid of the one and pretending the other still works as intended
was doubly irresponsible.
>> +(defun socks-open-network-stream (name buffer host service &rest params)
>> + "Open and return a connection, possibly proxied over SOCKS.
>
> The changes in this public function are so significant that I don't
> understand how they can be suggested so close to the branching.
The old signature was
(name buffer host service) -> process
and the new &rest arguments would be optional. And since the lone
in-tree call site sticks to the four required positionals, I didn't
think a move from (4 . 4) to (4 . many), in `func-arity' terms, stood to
break any advice in the wild. Still, there are side effects in the new
version that could use more thorough exploring, and further attention
could be paid to its treatment of `socks-override-functions' in terms of
preserving old behavior.
> If it is possible to add support for SOCKS 4a without affecting any
> previously supported versions, I'm fine. Adding tests is also fine.
> But for the rest, I think you should wait until after the release
> branch is cut and install this on the master branch. Sorry, it really
> is too late for such changes.
You're very gracious, but I think I've learned my lesson and will
refrain from pursuing any of these changes for Emacs 29. Apologies for
abusing your time and maintainerly patience (yet again).
[0000-v1-v2.diff (text/x-patch, attachment)]
[0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch (text/x-patch, attachment)]
[0002-30.0.50-Improve-SOCKS-error-handling-and-add-support.patch (text/x-patch, attachment)]
[0003-WIP-30.0.50-Simplify-network-stream-openers-in-socks.patch (text/x-patch, attachment)]
[0004-POC-30.0.50-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 29 Nov 2022 14:36:02 GMT)
Full text and
rfc822 format available.
Message #46 received at 53941 <at> debbugs.gnu.org (full text, mbox):
> From: "J.P." <jp <at> neverwas.me>
> Cc: 53941 <at> debbugs.gnu.org, gnuhacker <at> member.fsf.org, larsi <at> gnus.org
> Date: Tue, 29 Nov 2022 06:24:15 -0800
>
> Eli Zaretskii <eliz <at> gnu.org> writes:
>
> > I'm really uncomfortable with installing these changes before the release
> > branch is cut. The changes are hardly trivial, some controversial even to
> > my eyes, even though I'm no expert on network connections.
>
> Well, I myself am just about the furthest thing from (an expert), which
> certainly doesn't comport well with dropping rash changes at the
> eleventh hour. (That was rather disrespectful on my part, so shame on
> me.) As such, if it's easier to revisit this once things settle down,
> just ignore this email and I'll re-ping you sometime down the road.
Yes, please do. The branch is cut now, so if we agree on installing such
changes on master, it's now up to you when to post another version of these
changes with the requisite fixes.
> >> +(defun socks-open-network-stream (name buffer host service &rest params)
> >> + "Open and return a connection, possibly proxied over SOCKS.
> >
> > The changes in this public function are so significant that I don't
> > understand how they can be suggested so close to the branching.
>
> The old signature was
>
> (name buffer host service) -> process
I didn't mean the signature (which is OK), I meant the body. It is very
different from the previous version.
> > If it is possible to add support for SOCKS 4a without affecting any
> > previously supported versions, I'm fine. Adding tests is also fine.
> > But for the rest, I think you should wait until after the release
> > branch is cut and install this on the master branch. Sorry, it really
> > is too late for such changes.
>
> You're very gracious, but I think I've learned my lesson and will
> refrain from pursuing any of these changes for Emacs 29. Apologies for
> abusing your time and maintainerly patience (yet again).
No need to apologize, this is software development.
Thanks for working on these issues.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Wed, 06 Sep 2023 22:26:01 GMT)
Full text and
rfc822 format available.
Message #49 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Eli Zaretskii <eliz <at> gnu.org> writes:
>> > I'm really uncomfortable with installing these changes before the release
>> > branch is cut. The changes are hardly trivial, some controversial even to
>> > my eyes, even though I'm no expert on network connections.
>>
>> Well, I myself am just about the furthest thing from (an expert), which
>> certainly doesn't comport well with dropping rash changes at the
>> eleventh hour. (That was rather disrespectful on my part, so shame on
>> me.) As such, if it's easier to revisit this once things settle down,
>> just ignore this email and I'll re-ping you sometime down the road.
>
> Yes, please do. The branch is cut now, so if we agree on installing such
> changes on master, it's now up to you when to post another version of these
> changes with the requisite fixes.
Could we revisit this now? It sounds like something we'd want to fix.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Thu, 07 Sep 2023 05:55:02 GMT)
Full text and
rfc822 format available.
Message #52 received at 53941 <at> debbugs.gnu.org (full text, mbox):
> From: Stefan Kangas <stefankangas <at> gmail.com>
> Date: Wed, 6 Sep 2023 15:25:19 -0700
> Cc: "J.P." <jp <at> neverwas.me>, larsi <at> gnus.org, gnuhacker <at> member.fsf.org,
> 53941 <at> debbugs.gnu.org
>
> Eli Zaretskii <eliz <at> gnu.org> writes:
>
> >> > I'm really uncomfortable with installing these changes before the release
> >> > branch is cut. The changes are hardly trivial, some controversial even to
> >> > my eyes, even though I'm no expert on network connections.
> >>
> >> Well, I myself am just about the furthest thing from (an expert), which
> >> certainly doesn't comport well with dropping rash changes at the
> >> eleventh hour. (That was rather disrespectful on my part, so shame on
> >> me.) As such, if it's easier to revisit this once things settle down,
> >> just ignore this email and I'll re-ping you sometime down the road.
> >
> > Yes, please do. The branch is cut now, so if we agree on installing such
> > changes on master, it's now up to you when to post another version of these
> > changes with the requisite fixes.
>
> Could we revisit this now? It sounds like something we'd want to fix.
No objections from me, but I don't think we saw "another version"
posted. J.P.?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Thu, 07 Sep 2023 13:27:01 GMT)
Full text and
rfc822 format available.
Message #55 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Eli Zaretskii <eliz <at> gnu.org> writes:
>> From: Stefan Kangas <stefankangas <at> gmail.com>
>> Date: Wed, 6 Sep 2023 15:25:19 -0700
>> Cc: "J.P." <jp <at> neverwas.me>, larsi <at> gnus.org, gnuhacker <at> member.fsf.org,
>> 53941 <at> debbugs.gnu.org
>>
>> Eli Zaretskii <eliz <at> gnu.org> writes:
>>
>> >> > I'm really uncomfortable with installing these changes before the release
>> >> > branch is cut. The changes are hardly trivial, some controversial even to
>> >> > my eyes, even though I'm no expert on network connections.
>> >>
>> >> Well, I myself am just about the furthest thing from (an expert), which
>> >> certainly doesn't comport well with dropping rash changes at the
>> >> eleventh hour. (That was rather disrespectful on my part, so shame on
>> >> me.) As such, if it's easier to revisit this once things settle down,
>> >> just ignore this email and I'll re-ping you sometime down the road.
>> >
>> > Yes, please do. The branch is cut now, so if we agree on installing such
>> > changes on master, it's now up to you when to post another version of these
>> > changes with the requisite fixes.
>>
>> Could we revisit this now? It sounds like something we'd want to fix.
>
> No objections from me, but I don't think we saw "another version"
> posted. J.P.?
Hi, unfortunately, there is no other version of any real note, at least
not yet. The attached version is slightly updated but lacks substantial
progress in the areas that matter most. As of now, the only patches I'd
be comfortable offering would be the first two, which aren't even
directly related to this bug.
In addition to providing some cosmetic refactoring [1], the third patch
is mainly just a working placeholder for the socks.el side of a
hypothetical integration with the URL framework (currently imagined as
an overhauled version of `socks-open-connection' made newly compatible
with the `url-proxy' and `open-network-stream' interfaces). A sketch of
the URL side of such an arrangement can be found in the fourth patch,
but it isn't well thought out (or thoroughly researched).
AFAIR, the OP's main complaint concerns the lack of a transparent and
easily configurable experience for proxying built-in Emacs applications
over SOCKS, perhaps compared to what you'd get with a typical web
browser. I think we can probably agree that users shouldn't have to
customize both `socks'- and `url'-owned options or write extra code to
achieve a working setup, which is how things are currently (AFAICT).
If we're to solve this using the `url' library's existing public API, it
may need to be more clearly defined in a few areas. For example, at
least one function (`url-http-find-free-connection') uses the `host' and
`service' parameters of `url-open-stream' for dialing HTTP proxies. But
if other protocols are meant to do the same, they'll need to somehow
encode both proxy- and logical-endpoint addresses into those two params
or find some other means of conveying the same info. The fourth patch
currently uses `url-using-proxy' as a dynamic variable for this purpose,
but perhaps that's unwise. It's quite possible I'm breaking something.
There's also the issue of DNS lookups for verifying domain certs. Last I
looked, we can't run `nsm' checks without involving the system resolver,
which may be a deal breaker for the more privacy minded. If that's true,
we may want to find an acceptable way of cluing folks in to the
situation.
As thing stand, I haven't really invested enough in understanding how
`url' works to take the lead in planning a comprehensive integration
strategy (sorry). But, perhaps others with open bugs in the same area
[2] might be interested in lending some expertise or insight. Thanks.
[1] The third patch also attempts to untangle some of the hairiness
brought about by the `socks-override-functions' flag. The original
picture looks like:
s-o-f: `socks-override-functions', flag var
o-n-s: `open-network-stream', standard (non-SOCKS) opener
o-n-s*: `open-network-stream' with s--o-n-s as :around advice
s-o-c: `socks-open-connection', proxy (SOCKS) opener
s-o-n-s: `socks-open-network-stream', semi o-n-s compatible thin
wrapper around s--o-n-s
s--o-n-s: `socks--open-network-stream': workhorse with SOCKS/non
code paths
| s-o-f | entry point | no-route | route |
|-------+---------------------+----------+----------------|
| t | o-n-s* -> s--o-n-s | o-n-s | s-o-c -> o-n-s |
| t/nil | s-o-n-s -> s--o-n-s | o-n-s | s-o-c -> o-n-s |
I've changed the above to:
s--o-n-s: thin wrapper around `s-o-n-s'
s-o-n-s: o-n-s compatible workhorse with SOCKS/non code paths
| s-o-f | entry point | no-route | route |
|-------+-------------------------------+----------+----------------|
| t | o-n-s* -> s--o-n-s -> s-o-n-s | o-n-s | s-o-c -> o-n-s |
| t/nil | s-o-n-s | o-n-s | s-o-c -> o-n-s |
[2] Bug#62598: "29.0.60; url-https-proxy-connect doesn't support
multi-stage auth to proxies"
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62598
[0000-v2-v3.diff (text/x-patch, attachment)]
[0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch (text/x-patch, attachment)]
[0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch (text/x-patch, attachment)]
[0003-POC-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0004-POC-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Thu, 07 Sep 2023 13:48:01 GMT)
Full text and
rfc822 format available.
Message #58 received at 53941 <at> debbugs.gnu.org (full text, mbox):
"J.P." <jp <at> neverwas.me> writes:
> As of now, the only patches I'd be comfortable offering would be the
> first two, which aren't even directly related to this bug.
Thanks. The first two patches do add tests as well, so I can see some
value in installing them separately, perhaps even right now. Even more
so if it simplifies your work on the tasks you think are more important.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Fri, 08 Sep 2023 02:56:02 GMT)
Full text and
rfc822 format available.
Message #61 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Stefan Kangas <stefankangas <at> gmail.com> writes:
> "J.P." <jp <at> neverwas.me> writes:
>
>> As of now, the only patches I'd be comfortable offering would be the
>> first two, which aren't even directly related to this bug.
>
> Thanks. The first two patches do add tests as well, so I can see some
> value in installing them separately, perhaps even right now. Even more
> so if it simplifies your work on the tasks you think are more important.
OK, nice. I'd also like to add at least one test case that simulates a
realistic error condition (and maybe also a NEWS item for v4a, if that's
considered a feature). If no one else has thoughts regarding the first
two, I'll install them in a few days.
(Congrats on your appointment, BTW.)
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Fri, 08 Sep 2023 11:05:01 GMT)
Full text and
rfc822 format available.
Message #64 received at 53941 <at> debbugs.gnu.org (full text, mbox):
"J.P." <jp <at> neverwas.me> writes:
> OK, nice. I'd also like to add at least one test case that simulates a
> realistic error condition (and maybe also a NEWS item for v4a, if that's
> considered a feature). If no one else has thoughts regarding the first
> two, I'll install them in a few days.
Sounds good to me.
> (Congrats on your appointment, BTW.)
Thanks!
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Fri, 08 Sep 2023 13:30:02 GMT)
Full text and
rfc822 format available.
Message #67 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
"J.P." <jp <at> neverwas.me> writes:
> There's also the issue of DNS lookups for verifying domain certs. Last I
> looked, we can't run `nsm' checks without involving the system resolver,
> which may be a deal breaker for the more privacy minded. If that's true,
> we may want to find an acceptable way of cluing folks in to the
> situation.
Just a tiny update to the POC `url' integration stuff in case anyone
ever tries it. Previously, when connecting to a .onion domain over TLS,
e.g.,
https://www.dwnewsgngmhlplxy6o2twtfgjnrnjxbegbwqx6wnotdhkzt562tszfid.onion/en/
you'd get spammed with "name or service unknown" messages in the echo
area with every EWW link clicked (at least on GNU Linux, probably from
something like GAI EAI_NONAME). Anyway, no longer. Also added a test, a
news entry, and some doc tweaks. Thanks.
[0000-v3-v4.diff (text/x-patch, attachment)]
[0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch (text/x-patch, attachment)]
[0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch (text/x-patch, attachment)]
[0003-POC-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0004-POC-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Sat, 09 Sep 2023 14:06:02 GMT)
Full text and
rfc822 format available.
Message #70 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I seem to have missed the fact that `url-methods' populates the option
`url-proxy-services' from environment variables when initializing
`url-scheme-registry'. As explained in the docs, it actually accepts
full URLs instead of just host:port pairs. I take this to mean it's
probably less disruptive than initially thought to extend this liberty
to `url-proxy-services' itself. I've updated the POC patches to reflect
this, so it should now be possible to do:
(setenv "HTTPS_PROXY" "socks5h://localhost:9050")
(eww "https://check.torproject.org/")
Of course, this shouldn't interfere with traditional http proxies, such
as those that provide CONNECT tunneling:
$ ncat -l --proxy-type http localhost 8888
(setenv "HTTPS_PROXY" "localhost:8888")
(eww "https://www.example.com/")
[0000-v4-v5.diff (text/x-patch, attachment)]
[0001-Don-t-hard-code-server-ports-in-SOCKS-tests.patch (text/x-patch, attachment)]
[0002-Improve-SOCKS-error-handling-and-add-support-for-4a.patch (text/x-patch, attachment)]
[0003-POC-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0004-POC-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Wed, 18 Oct 2023 13:41:01 GMT)
Full text and
rfc822 format available.
Message #73 received at 53941 <at> debbugs.gnu.org (full text, mbox):
"J.P." <jp <at> neverwas.me> writes:
> Stefan Kangas <stefankangas <at> gmail.com> writes:
>
>> "J.P." <jp <at> neverwas.me> writes:
>>
>>> As of now, the only patches I'd be comfortable offering would be the
>>> first two, which aren't even directly related to this bug.
>>
>> Thanks. The first two patches do add tests as well, so I can see some
>> value in installing them separately, perhaps even right now. Even more
>> so if it simplifies your work on the tasks you think are more important.
>
> OK, nice. I'd also like to add at least one test case that simulates a
> realistic error condition (and maybe also a NEWS item for v4a, if that's
> considered a feature). If no one else has thoughts regarding the first
> two, I'll install them in a few days.
s/days/fortnights/
(Apologies for the delay. Done now.)
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 19 Dec 2023 16:31:01 GMT)
Full text and
rfc822 format available.
Message #76 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
In light of recent related activity on the development mailing list [1],
I think it's worth summarizing where this bug stands, even though that
discussion seems to favor other approaches that I personally have little
interest in. As a reminder, this bug is about integrating the built-in
`socks' and `url' libraries to make proxying specific connections over
Tor mostly transparent and hassle free.
As it stands, the two biggest blockers I can see are both heavy on the
legwork side of things. The first and more daunting involves surveying
how the `url' library is used in the wild when it comes to proxies: what
interfaces people use and how that compares with what's prescribed in
the manual and what was promised in the original design discussions. The
goal would be to use these findings to enhance or overhaul the existing
interface and behavior and to affirm the result as canon. This will
likely involve reworking parts of the existing documentation and
saturating affected parts of the library in comprehensive test coverage.
It may also require central coordination to ensure interoperability
among consuming libraries and continuity of purpose among future
contributors.
The second and IMO easier (but still labor-intensive) task is fleshing
out and fortifying the actual networking implementation. What I propose
is that we start by leveraging the built-in logging facility of torsocks
itself [2]. By default, the verbose-level output is noisy but tolerable
after minimal modifications (e.g., by commenting out the "close" and
"fclose" statements). We'd need volunteers with access to the various
platforms Emacs supports (capable of running a Tor daemon) to run
through some contrived recipes, such as loading a site with `eww' and
fetching a package from an ELPA endpoint.
$ LD_PRELOAD=/home/me/torsocks/src/lib/.libs/libtorsocks.so \
TORSOCKS_LOG_FILE_PATH=/tmp/torsocks.log \
TORSOCKS_LOG_LEVEL=5 ./src/emacs -Q
What we'd be looking for in the output is activity in those libc "GAI"
functions shadowed by the program.
While this methodology seems hokey, I think it's actually preferable (at
least as a starting point) over more traditional tracing solutions
because with the latter we'd still have to isolate the set of
torsocks-shadowed functions in whatever recording is produced, and then
filter out all the false positives from things like connect(2) calls for
AF_LOCAL/AF_UNIX, which we don't care about. Moreover, these
hand-crafted logs show us other niceties, like parameters of interest
(socket type, hint flags, etc.), which makes sense because this program
comes from the Tor Project itself and is well written. Anyway, once we
have a solid idea of what needs intercepting and/or inhibiting [3],
we'll need volunteers yet again, this time to help run packet traces
against a prototype both for the proxied connection and for DNS leakage.
Obviously, help from those familiar with the Emacs network stack would
go a long way here.
Anyway, the attached POC implementation is basically just a slightly
cleaned up version of the same patch set I've been fiddling with this
entire bug thread. WRT to the `url' side, I've done basically zero
additional research, and the interface you see is just a "best guess"
based on the current documentation and a cursory stroll around the
library. As for the `socks' side, everything is based on observations
from exactly one Emacs installation on one GNU/Linux machine. That it
doesn't leak DNS (for me) should just be a data point.
If others want to work on this or take over, please let me know. I
encourage anyone with a similar setup (GNU/Linux, x86_64, modern
libraries) to try them out. You'll need a Tor service running locally,
preferably with the default configuration. The API is pretty basic and
could be simplified even further. For now, it's
(require 'socks)
(require 'url)
(setopt url-proxy-services '(("https" . "socks5h://localhost:9050")
("http" . "socks5h://localhost:9050"))
socks-server '("tor" "localhost" 9050 5)
socks-username ""
socks-password "")
then something like
M-x eww RET https://check.torproject.org RET
or
M-x list-packages RET.
[1] https://lists.gnu.org/archive/html/emacs-devel/2023-12/msg00570.html
[2] https://gitlab.torproject.org/tpo/core/torsocks.git
[3] Just to clarify a couple things I've personally found confusing,
in case others may find them beneficial:
If we only decide to support local proxies, we don't have to worry
about DNS leakage for the proxy service itself. I've heard folks
express concern about calls to getaddrinfo and, in particular, the
async `:nowait' variant getaddrinfo_a (because torsocks AFAIK
doesn't shadow it). But those won't come into play since our only
connections through `make-network-process' are to the local proxy
service. IOW, the actual remote lookup is tunneled, so the only
calls we'll need to care about intercepting WRT DNS are those from
`nsm-check' to `network-lookup-address-info'. (In the attached POC
implementation, I simply reroute these through a tor-specific
resolver.)
The "onion cookie" bookkeeping business performed by torsocks is of
no interest to us because we control the means of connection. IOW,
at no point should an onion address (or that of a normal remote
endpoint) be exposed to the underlying networking machinery. It's
all just part of the opaque application data unit (payload).
[0000-v5-v6.diff (text/x-patch, attachment)]
[0001-POC-Support-SOCKS-resolve-extension.patch (text/x-patch, attachment)]
[0002-POC-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0003-POC-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Fri, 23 Aug 2024 21:48:01 GMT)
Full text and
rfc822 format available.
Message #79 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Hello all. I don't pretend to track most of what is going on in this bug thread, but I was wanting to draw more attention to the specific issue of proxies and certs. I use Emacs Elpher for gemini browsing. As a privacy minded individual, I want to by default route everything, including DNS, through my local TOR proxy (localhost:9050). But I also want to be able to check and approve capsule certs and, especially, cert changes. But in Elpher, I have to pick one or the other, since turning on the proxy disables the cert checks. As it has been explained to me, this is due to the current situation with nsm, where you can't have `nsm' checks without also leaking DNS.
--
Christopher Howard
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Sat, 14 Sep 2024 13:35:02 GMT)
Full text and
rfc822 format available.
Message #82 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Christopher Howard <christopher <at> librehacker.com> writes:
> Hello all. I don't pretend to track most of what is going on in this bug thread, but I was wanting to draw more attention to the specific issue of proxies and certs. I use Emacs Elpher for gemini browsing. As a privacy minded individual, I want to by default route everything, including DNS, through my local TOR proxy (localhost:9050). But I also want to be able to check and approve capsule certs and, especially, cert changes. But in Elpher, I have to pick one or the other, since turning on the proxy disables the cert checks. As it has been explained to me, this is due to the current situation with nsm, where you can't have `nsm' checks without also leaking DNS.
Is there an open bug report for leaking DNS with a tor proxy+nsm?
If not, would you be willing to report one (including all the details)?
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 16 Sep 2024 02:00:01 GMT)
Full text and
rfc822 format available.
Message #85 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Stefan Kangas <stefankangas <at> gmail.com> writes:
> Christopher Howard <christopher <at> librehacker.com> writes:
>
>> Hello all. I don't pretend to track most of what is going on in this bug
>> thread, but I was wanting to draw more attention to the specific issue of
>> proxies and certs. I use Emacs Elpher for gemini browsing. As a privacy
>> minded individual, I want to by default route everything, including DNS,
>> through my local TOR proxy (localhost:9050). But I also want to be able to
>> check and approve capsule certs and, especially, cert changes. But in
>> Elpher, I have to pick one or the other, since turning on the proxy disables
>> the cert checks. As it has been explained to me, this is due to the current
>> situation with nsm, where you can't have `nsm' checks without also leaking
>> DNS.
>
> Is there an open bug report for leaking DNS with a tor proxy+nsm?
Not that I'm aware of.
>
> If not, would you be willing to report one (including all the details)?
In a sense, the issue only exists in the context of trying to integrate
`socks' with other libraries like `nsm' and `url' (this bug). As such,
there's currently no high-level way (I can think of) to demonstrate its
presence. For that, you'd need an app like Elpher to support connecting
to TLS-terminated endpoints through a SOCKS proxy while verifying them
with `nsm' checks. And you'd need to eavesdrop on it doing so in a
controlled environment where DNS lookups are well understood.
To see how something nearer to a proper (though limited) integration
_could_ work, you can try the demo in the log message of the last of the
attached PoC patches (0004). While it "works," it's quite brittle in the
sense that any unsupported but otherwise normal config patterns (e.g.,
:nowait t) or any related but undetected change to an affected library
(or the underlying networking stack) could render the whole thing bunk.
As I've struggled to explain up thread, the DNS leakage issue is larger
than any prospective integration, `nsm' or otherwise. But, for the sake
of discussion, if we were to zoom in on that library in particular, the
reason for the leakage should be pretty clear. AFAICT, the function
`nsm-should-check' always performs a lookup in order to support the
`nsm-trust-local-network' feature (original author Robert Cc'd). One
possible workaround might be to rework the function slightly to prevent
that, as shown in the first of the attached patches (0001).
Anyway, to truly tackle this issue, I still contend we'd need to
intercept calls to any glibc GAI-related functions and gate them with
some kind of async-friendly mechanism (perhaps a process property) that
suppresses their invocation for the lifetime of the process. The API
could be as simple as:
(make-network-process ... :nolookup t ...)
But for this, we'd surely need help from someone familiar with that part
of Emacs.
Thanks.
[0000-v6-v7.diff (text/x-patch, attachment)]
[0001-Only-conditionally-resolve-hosts-in-nsm-should-check.patch (text/x-patch, attachment)]
[0002-POC-Support-SOCKS-resolve-extension.patch (text/x-patch, attachment)]
[0003-POC-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0004-POC-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Mon, 16 Sep 2024 13:36:01 GMT)
Full text and
rfc822 format available.
Message #88 received at 53941 <at> debbugs.gnu.org (full text, mbox):
>>>>> On Sun, 15 Sep 2024 18:59:10 -0700, "J.P." <jp <at> neverwas.me> said:
JP> As I've struggled to explain up thread, the DNS leakage issue is larger
JP> than any prospective integration, `nsm' or otherwise. But, for the sake
JP> of discussion, if we were to zoom in on that library in particular, the
JP> reason for the leakage should be pretty clear. AFAICT, the function
JP> `nsm-should-check' always performs a lookup in order to support the
JP> `nsm-trust-local-network' feature (original author Robert Cc'd). One
JP> possible workaround might be to rework the function slightly to prevent
JP> that, as shown in the first of the attached patches (0001).
More information hiding by default is a good thing. (Iʼm not the
original author, I just changed it to look at the actual local
addresses instead of hardcoding them)
JP> Anyway, to truly tackle this issue, I still contend we'd need to
JP> intercept calls to any glibc GAI-related functions and gate them with
JP> some kind of async-friendly mechanism (perhaps a process property) that
JP> suppresses their invocation for the lifetime of the process. The API
JP> could be as simple as:
JP> (make-network-process ... :nolookup t ...)
Iʼm not sure what suppressing DNS lookups would get us apart from more
failure modes, but I havenʼt thought about it deeply.
JP> But for this, we'd surely need help from someone familiar with that part
JP> of Emacs.
JP> * lisp/net/nsm.el (nsm-should-check): Rework in a functionally
JP> equivalent way, except forgo calling both `network-lookup-address-info'
JP> and `network-interface-list' unless the various conditions regarding
JP> `nsm-trust-local-network' are first satisfied. Replace `mapc' with
JP> `dolist' to align with modern sensibilities. (Bug#53941)
Careful now, somebody even more modern might come along and replace `dolist' with
`seq-do' ☺️
JP> ---
JP> lisp/net/nsm.el | 33 ++++++++++++---------------------
JP> 1 file changed, 12 insertions(+), 21 deletions(-)
JP> diff --git a/lisp/net/nsm.el b/lisp/net/nsm.el
JP> index e8fdb9b183b..a8a3abb6a2d 100644
JP> --- a/lisp/net/nsm.el
JP> +++ b/lisp/net/nsm.el
JP> @@ -226,27 +226,18 @@ nsm-should-check
JP> host address is a localhost address, or in the same subnet as one
JP> of the local interfaces, this function returns nil. Non-nil
JP> otherwise."
JP> - (let ((addresses (network-lookup-address-info host))
JP> - (network-interface-list (network-interface-list t))
JP> - (off-net t))
JP> - (when
JP> - (or (and (functionp nsm-trust-local-network)
JP> - (funcall nsm-trust-local-network))
JP> - nsm-trust-local-network)
JP> - (mapc
JP> - (lambda (ip)
JP> - (mapc
JP> - (lambda (info)
JP> - (let ((local-ip (nth 1 info))
JP> - (mask (nth 3 info)))
JP> - (when
JP> - (nsm-network-same-subnet (substring local-ip 0 -1)
JP> - (substring mask 0 -1)
JP> - (substring ip 0 -1))
JP> - (setq off-net nil))))
JP> - network-interface-list))
JP> - addresses))
JP> - off-net))
JP> + (not (and-let* (((or (and (functionp nsm-trust-local-network)
JP> + (funcall nsm-trust-local-network))
JP> + nsm-trust-local-network))
JP> + (addresses (network-lookup-address-info host))
JP> + (network-interface-list (network-interface-list t)))
JP> + (catch 'off-net
JP> + (dolist (ip addresses)
JP> + (dolist (info network-interface-list)
JP> + (when (nsm-network-same-subnet (substring (nth 1 info) 0 -1)
JP> + (substring (nth 3 info) 0 -1)
JP> + (substring ip 0 -1))
JP> + (throw 'off-net t))))))))
Since youʼve inverted the test, you should probably invert the name of
`off-net'.
Robert
--
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 17 Sep 2024 01:53:02 GMT)
Full text and
rfc822 format available.
Message #91 received at 53941 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Robert Pluim <rpluim <at> gmail.com> writes:
> More information hiding by default is a good thing. (Iʼm not the
> original author, I just changed it to look at the actual local
> addresses instead of hardcoding them)
D'oh, I see that now. The original author was one Mr. Wong. If you'd
like to be spared any further spam related to this bug, please say so,
and I'll remove you from the Cc's. (But if not, I could certainly use
the input.)
> JP> The API could be as simple as:
>
> JP> (make-network-process ... :nolookup t ...)
>
> Iʼm not sure what suppressing DNS lookups would get us apart from more
> failure modes, but I havenʼt thought about it deeply.
Hm, right. I suppose doing that would mostly be useless for this type of
proxy because the :host property of the process is usually the one being
looked up, and if the `socks' side is handling things properly, :host
should only ever be the proxy server itself. Also, the lookups I was
hoping to prevent (or redirect through Tor with something like the
attached 0002 PoC patch) would need to be limited to only a specific
application rather than all of Emacs, which seems rather unrealistic.
> JP> * lisp/net/nsm.el (nsm-should-check): Rework in a functionally
> JP> equivalent way, except forgo calling both `network-lookup-address-info'
> JP> and `network-interface-list' unless the various conditions regarding
> JP> `nsm-trust-local-network' are first satisfied. Replace `mapc' with
> JP> `dolist' to align with modern sensibilities. (Bug#53941)
>
> Careful now, somebody even more modern might come along and replace `dolist' with
> `seq-do' ☺️
Good point. I'll be sure and use `brat-do' from now on, just to be safe.
(Hopefully, you have no idea what that means.)
> JP> + (not (and-let* (((or (and (functionp nsm-trust-local-network)
> JP> + (funcall nsm-trust-local-network))
> JP> + nsm-trust-local-network))
> JP> + (addresses (network-lookup-address-info host))
> JP> + (network-interface-list (network-interface-list t)))
> JP> + (catch 'off-net
> JP> + (dolist (ip addresses)
> JP> + (dolist (info network-interface-list)
> JP> + (when (nsm-network-same-subnet (substring (nth 1 info) 0 -1)
> JP> + (substring (nth 3 info) 0 -1)
> JP> + (substring ip 0 -1))
> JP> + (throw 'off-net t))))))))
>
> Since youʼve inverted the test, you should probably invert the name of
> `off-net'.
Ah, took that "by rote" from the old sentinel variable, but as you say,
it doesn't comport with the semantics. Changed to `nsm-should-check'.
Overall, I'll have to think on this bug a bit more. If Christopher or
the Elpher people want this specific workaround in tree, I suppose we
can accommodate, but I doubt there's any rush.
Cheers.
[0000-v7-v8.diff (text/x-patch, attachment)]
[0001-Only-conditionally-resolve-hosts-in-nsm-should-check.patch (text/x-patch, attachment)]
[0002-POC-Support-SOCKS-resolve-extension.patch (text/x-patch, attachment)]
[0003-POC-Simplify-network-stream-openers-in-socks.el.patch (text/x-patch, attachment)]
[0004-POC-Integrate-the-socks-and-url-libraries.patch (text/x-patch, attachment)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 17 Sep 2024 07:31:02 GMT)
Full text and
rfc822 format available.
Message #94 received at 53941 <at> debbugs.gnu.org (full text, mbox):
>>>>> On Mon, 16 Sep 2024 18:52:04 -0700, "J.P." <jp <at> neverwas.me> said:
JP> Robert Pluim <rpluim <at> gmail.com> writes:
>> More information hiding by default is a good thing. (Iʼm not the
>> original author, I just changed it to look at the actual local
>> addresses instead of hardcoding them)
JP> D'oh, I see that now. The original author was one Mr. Wong. If you'd
JP> like to be spared any further spam related to this bug, please say so,
JP> and I'll remove you from the Cc's. (But if not, I could certainly use
JP> the input.)
I donʼt mind.
JP> + (not (and-let* (((or (and (functionp nsm-trust-local-network)
JP> + (funcall nsm-trust-local-network))
JP> + nsm-trust-local-network))
JP> + (addresses (network-lookup-address-info host))
JP> + (network-interface-list (network-interface-list t)))
JP> + (catch 'off-net
JP> + (dolist (ip addresses)
JP> + (dolist (info network-interface-list)
JP> + (when (nsm-network-same-subnet (substring (nth 1 info) 0 -1)
JP> + (substring (nth 3 info) 0 -1)
JP> + (substring ip 0 -1))
JP> + (throw 'off-net t))))))))
>>
>> Since youʼve inverted the test, you should probably invert the name of
>> `off-net'.
JP> Ah, took that "by rote" from the old sentinel variable, but as you say,
JP> it doesn't comport with the semantics. Changed to `nsm-should-check'.
OK. I think you can push that patch to master if Eli is ok with it,
regardless of what happens with the others.
Robert
--
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 17 Sep 2024 12:43:01 GMT)
Full text and
rfc822 format available.
Message #97 received at 53941 <at> debbugs.gnu.org (full text, mbox):
> From: Robert Pluim <rpluim <at> gmail.com>
> Cc: Stefan Kangas <stefankangas <at> gmail.com>, Christopher Howard
> <christopher <at> librehacker.com>, 53941 <at> debbugs.gnu.org, larsi <at> gnus.org,
> Eli Zaretskii <eliz <at> gnu.org>, gnuhacker <at> member.fsf.org
> Date: Tue, 17 Sep 2024 09:29:15 +0200
>
>
> OK. I think you can push that patch to master if Eli is ok with it,
> regardless of what happens with the others.
I know nothing about socks and tor, so feel free to act on your own
judgment in this matter.
Thanks.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Tue, 17 Sep 2024 13:56:02 GMT)
Full text and
rfc822 format available.
Message #100 received at 53941 <at> debbugs.gnu.org (full text, mbox):
>>>>> On Tue, 17 Sep 2024 15:41:33 +0300, Eli Zaretskii <eliz <at> gnu.org> said:
Eli> I know nothing about socks and tor, so feel free to act on your own
Eli> judgment in this matter.
Eli> Thanks.
I think JP has commit privileges, otherwise I can push it.
Robert
--
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#53941; Package
emacs.
(Wed, 18 Sep 2024 01:12:01 GMT)
Full text and
rfc822 format available.
Message #103 received at 53941 <at> debbugs.gnu.org (full text, mbox):
Robert Pluim <rpluim <at> gmail.com> writes:
>>>>>> On Tue, 17 Sep 2024 15:41:33 +0300, Eli Zaretskii <eliz <at> gnu.org> said:
>
> Eli> I know nothing about socks and tor, so feel free to act on your own
> Eli> judgment in this matter.
>
> Eli> Thanks.
>
> I think JP has commit privileges, otherwise I can push it.
Installed on master as
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=50deb59a
Leaving the bug open because ^ only dealt with an ancillary issue.
Thanks.
This bug report was last modified 274 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.