GNU bug report logs - #73199
url-retrieve is not thread-safe

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 73199 in the body.
You can then email your comments to 73199 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: bug-gnu-emacs <at> gnu.org
Subject: soap-client; soap-invoke-internal is not thread-safe
Date: Thu, 12 Sep 2024 16:20:03 +0200

Forward to debbugs

From: Michael Albinus <michael.albinus <at> gmx.de>
Subject: soap-client; soap-invoke-internal is not thread-safe
To: Alexandru Harsanyi <AlexHarsanyi <at> gmail.com>
Date: Thu, 12 Sep 2024 15:57:45 +0200 (19 minutes, 24 seconds ago)

In package debbugs, I try to use Emacs Lisp threads when retrieving SOAP
data from the debbugs server. However, ocaasionally I run into the error

--8<---------------cut here---------------start------------->8---
(error "Attempt to accept output from process debbugs.gnu.org locked to thread #<thread 0xc288a0>")
--8<---------------cut here---------------end--------------->8---

My work around is to advice url-http-create-request

--8<---------------cut here---------------start------------->8---
  (advice-add
   'url-http-create-request :around
   (lambda (orig-fun)
     "Set `url-http-attempt-keepalives' to nil."
     (setq url-http-attempt-keepalives nil)
     (funcall orig-fun))
   '(name debbugs-advice))
--8<---------------cut here---------------end--------------->8---

However, it would be great if soap-invoke-internal could care.

Emacs  : GNU Emacs 31.0.50 (build 35, x86_64-pc-linux-gnu, GTK+ Version 3.24.43, cairo version 1.18.0)
 of 2024-09-12
Package: soap-client

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
To: Michael Albinus via "Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs <at> gnu.org>
Cc: Alexandru Harsanyi <AlexHarsanyi <at> gmail.com>,
 Michael Albinus <michael.albinus <at> gmx.de>, 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: soap-client; soap-invoke-internal is not thread-safe
Date: Sat, 14 Sep 2024 10:06:47 -0400

Hi Michael,

Michael Albinus writes:

> Forward to debbugs
>
> From: Michael Albinus <michael.albinus <at> gmx.de>
> Subject: soap-client; soap-invoke-internal is not thread-safe
> To: Alexandru Harsanyi <AlexHarsanyi <at> gmail.com>
> Date: Thu, 12 Sep 2024 15:57:45 +0200 (19 minutes, 24 seconds ago)
>
> In package debbugs, I try to use Emacs Lisp threads when retrieving SOAP
> data from the debbugs server. However, ocaasionally I run into the error
>
> --8<---------------cut here---------------start------------->8---
> (error "Attempt to accept output from process debbugs.gnu.org locked to thread #<thread 0xc288a0>")
> --8<---------------cut here---------------end--------------->8---
>
> My work around is to advice url-http-create-request
>
> --8<---------------cut here---------------start------------->8---
>   (advice-add
>    'url-http-create-request :around
>    (lambda (orig-fun)
>      "Set `url-http-attempt-keepalives' to nil."
>      (setq url-http-attempt-keepalives nil)
>      (funcall orig-fun))
>    '(name debbugs-advice))
> --8<---------------cut here---------------end--------------->8---
>
> However, it would be great if soap-invoke-internal could care.

I am not able to test Excorporate's usage of soap-client anymore to
check if a candidate patch would break anything.  I will ask on
emacs-devel if anyone wants to take over maintenance of Excorporate and
its dependencies.

A starting point for adding thread safety would be a minimal test case
that calls soap-invoke-internal against the debbugs server from multiple
different threads.  Are you able to extract such a test case?

If the resulting patch is obviously safe for single-threaded usage by
Excorporate, then I could push it to soap-client.el.

Thanks,
Thomas

Message #14 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Thomas Fitzsimmons <fitzsim <at> fitzsim.org>
Cc: "Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of
 text editors" <bug-gnu-emacs <at> gnu.org>,
 Alexandru Harsanyi <AlexHarsanyi <at> gmail.com>, 73199 <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#73199: soap-client; soap-invoke-internal is not thread-safe
Date: Fri, 20 Sep 2024 11:58:13 +0200

Thomas Fitzsimmons <fitzsim <at> fitzsim.org> writes:

> Hi Michael,

Hi Thomas,

sorry for the late reply, I was out of order for a week.

> A starting point for adding thread safety would be a minimal test case
> that calls soap-invoke-internal against the debbugs server from multiple
> different threads.  Are you able to extract such a test case?

Install the just-released debbugs 0.41. Open a new Emacs session, load
debbugs-gnu.

Deactivate the advice I have added to mitigate the problem.

--8<---------------cut here---------------start------------->8---
(defalias 'debbugs-compat-add-debbugs-advice 'ignore)
(defalias 'debbugs-compat-remove-debbugs-advice 'ignore)
--8<---------------cut here---------------end--------------->8---

Retrieve the most recent 10 bugs. This happens in the main thread.

--8<---------------cut here---------------start------------->8---
M-x debbugs-gnu-bugs RET RET
--8<---------------cut here---------------end--------------->8---

Retrieve the most recent 100 bugs. This happens in another thread.

--8<---------------cut here---------------start------------->8---
M-x debbugs-gnu-bugs RET -100 RET
--8<---------------cut here---------------end--------------->8---

You'll see the error.

--8<---------------cut here---------------start------------->8---
Error #<thread debbugs>: (error "Attempt to accept output from process debbugs.gnu.org locked to thread #<thread 0xc288a0>")
--8<---------------cut here---------------end--------------->8---

BTW, in a short discussion on emacs-devel, Stefan Monnier proposed to
fix this problem in url-http.el. Don't know what's the better approach.

> Thanks,
> Thomas

Best regards, Michael.

Message #22 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: 73199 <at> debbugs.gnu.org
Cc: AlexHarsanyi <at> gmail.com, Thomas Fitzsimmons <fitzsim <at> fitzsim.org>,
 monnier <at> iro.umontreal.ca
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Mon, 30 Sep 2024 17:12:34 +0200

[Message part 1 (text/plain, inline)]

Michael Albinus <michael.albinus <at> gmx.de> writes:

Hi,

> BTW, in a short discussion on emacs-devel, Stefan Monnier proposed to
> fix this problem in url-http.el. Don't know what's the better approach.

I gave it a try. The appended patch seems to fix the problem. At least
in my use case, there's no error anymore when I use debbugs w/o the
advice in debbugs-compat.el.

Comments?

Best regards, Michael.

[Message part 2 (text/x-patch, attachment)]

Message #25 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Mon, 30 Sep 2024 18:40:42 +0300

> Cc: AlexHarsanyi <at> gmail.com, Thomas Fitzsimmons <fitzsim <at> fitzsim.org>,
>  monnier <at> iro.umontreal.ca
> Date: Mon, 30 Sep 2024 17:12:34 +0200
> From:  Michael Albinus via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> > BTW, in a short discussion on emacs-devel, Stefan Monnier proposed to
> > fix this problem in url-http.el. Don't know what's the better approach.
> 
> I gave it a try. The appended patch seems to fix the problem. At least
> in my use case, there's no error anymore when I use debbugs w/o the
> advice in debbugs-compat.el.
> 
> Comments?

ENOPATCH

Message #28 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Mon, 30 Sep 2024 18:42:55 +0200

[Message part 1 (text/plain, inline)]

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Comments?
>
> ENOPATCH

Oops, sorry. I've appended the vc-dir buffer, not the vc-diff ...

[Message part 2 (text/x-patch, attachment)]

Message #31 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Mon, 30 Sep 2024 20:38:49 +0300

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: 73199 <at> debbugs.gnu.org,  AlexHarsanyi <at> gmail.com,  fitzsim <at> fitzsim.org,
>   monnier <at> iro.umontreal.ca
> Date: Mon, 30 Sep 2024 18:42:55 +0200
> 
> Oops, sorry. I've appended the vc-dir buffer, not the vc-diff ...

Thanks.  I'd appreciate if you could explain why url-retrieve is
currently not thread-safe, and how this patch is supposed to solve
that.

Message #34 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Wed, 02 Oct 2024 18:34:55 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Michael Albinus <michael.albinus <at> gmx.de>
>> Cc: 73199 <at> debbugs.gnu.org,  AlexHarsanyi <at> gmail.com,  fitzsim <at> fitzsim.org,
>>   monnier <at> iro.umontreal.ca
>> Date: Mon, 30 Sep 2024 18:42:55 +0200
>>
>> Oops, sorry. I've appended the vc-dir buffer, not the vc-diff ...
>
> Thanks.  I'd appreciate if you could explain why url-retrieve is
> currently not thread-safe, and how this patch is supposed to solve
> that.

In <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=73199#14>, there's a
recipe for reproduction.

The general problem is when you want to access the same HTTP server from
a different thread, and url-http-attempt-keepalives is set to t. This
happens, for example, in debbugs 0.41.

Let's assume, there is an access to debbugs:443 from a given thread, for
example the main thread. Soap-client serves. It also binds
url-http-attempt-keepalives to t, the https process stays alive.

Later on, there is an access to debbugs:443 from another thread. It
checks, whether there is a running process for that target, and tries to
reuse. We get the error

--8<---------------cut here---------------start------------->8---
Error #<thread debbugs>: (error "Attempt to accept output from process debbugs.gnu.org locked to thread #<thread 0xc288a0>")
--8<---------------cut here---------------end--------------->8---

My patch tries to fix this. The list of still open processes is the hash
table url-http-open-connections. It has the ky (cons host port), and as
value a list of processes.

My patch changes this such a way, that the key is now (list host port
thread). Therefore, only open processes which belong to the same thread
are checked.

This seems to work sufficiently.

Best regards, Michael.

Message #37 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Wed, 02 Oct 2024 21:30:11 +0300

Message #40 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Thu, 10 Oct 2024 13:29:42 +0200

Michael Albinus <michael.albinus <at> gmx.de> writes:

> Eli Zaretskii <eliz <at> gnu.org> writes:
>
>>> From: Michael Albinus <michael.albinus <at> gmx.de>
>>> Cc: 73199 <at> debbugs.gnu.org,  AlexHarsanyi <at> gmail.com,  fitzsim <at> fitzsim.org,
>>>   monnier <at> iro.umontreal.ca
>>> Date: Mon, 30 Sep 2024 18:42:55 +0200
>>>
>>> Oops, sorry. I've appended the vc-dir buffer, not the vc-diff ...
>>
>> Thanks.  I'd appreciate if you could explain why url-retrieve is
>> currently not thread-safe, and how this patch is supposed to solve
>> that.
>
> In <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=73199#14>, there's a
> recipe for reproduction.
>
> The general problem is when you want to access the same HTTP server from
> a different thread, and url-http-attempt-keepalives is set to t. This
> happens, for example, in debbugs 0.41.
>
> Let's assume, there is an access to debbugs:443 from a given thread, for
> example the main thread. Soap-client serves. It also binds
> url-http-attempt-keepalives to t, the https process stays alive.
>
> Later on, there is an access to debbugs:443 from another thread. It
> checks, whether there is a running process for that target, and tries to
> reuse. We get the error
>
> Error #<thread debbugs>: (error "Attempt to accept output from process debbugs.gnu.org locked to thread #<thread 0xc288a0>")
>
> My patch tries to fix this. The list of still open processes is the hash
> table url-http-open-connections. It has the ky (cons host port), and as
> value a list of processes.
>
> My patch changes this such a way, that the key is now (list host port
> thread). Therefore, only open processes which belong to the same thread
> are checked.
>
> This seems to work sufficiently.

Ping. Any comments? Otherwise, I'll install the patch in a couple of days.

Best regards, Michael.

Message #43 received at 73199 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199 <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Thu, 10 Oct 2024 16:10:29 +0300

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: AlexHarsanyi <at> gmail.com,  fitzsim <at> fitzsim.org,  monnier <at> iro.umontreal.ca,
>   73199 <at> debbugs.gnu.org
> Date: Thu, 10 Oct 2024 13:29:42 +0200
> 
> Ping. Any comments? Otherwise, I'll install the patch in a couple of days.

Sorry, too much stuff on my table.

Please go ahead and install, I think the patch is okay.  (I presume
you know about set-process-thread, and that it doesn't help in this
scenario?)

Message #48 received at 73199-done <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: AlexHarsanyi <at> gmail.com, fitzsim <at> fitzsim.org, monnier <at> iro.umontreal.ca,
 73199-done <at> debbugs.gnu.org
Subject: Re: bug#73199: url-retrieve is not thread-safe
Date: Fri, 11 Oct 2024 12:44:14 +0200

Version: 31.1

Eli Zaretskii <eliz <at> gnu.org> writes:

Hi Eli,

> Please go ahead and install, I think the patch is okay.

Done. I'm closing the bug, therefore.

> (I presume you know about set-process-thread, and that it doesn't help
> in this scenario?)

Yes, I know it. But I cannot use it; the process is already in the
proper thread where it should be.

Best regards, Michael.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.