From unknown Sat Jun 14 19:33:49 2025 X-Loop: help-debbugs@gnu.org Subject: bug#31879: Caching in the "module import obarray" is not thread-safe Resent-From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Mon, 18 Jun 2018 13:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 31879 X-GNU-PR-Package: guile X-GNU-PR-Keywords: To: 31879@debbugs.gnu.org X-Debbugs-Original-To: bug-guile@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.15293297307360 (code B ref -1); Mon, 18 Jun 2018 13:49:02 +0000 Received: (at submit) by debbugs.gnu.org; 18 Jun 2018 13:48:50 +0000 Received: from localhost ([127.0.0.1]:54504 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUuWY-0001ue-0Y for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:50 -0400 Received: from eggs.gnu.org ([208.118.235.92]:42011) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUuWW-0001uS-Ub for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUuWQ-00054Q-SU for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:43 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:39829) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fUuWQ-00054M-Om for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:32977) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUuWM-0002K5-JX for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUuWJ-0004v8-Il for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:38 -0400 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38344) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUuWJ-0004ur-Ez for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:35 -0400 Received: from [193.50.110.191] (port=59408 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1fUuWJ-0001jF-23 for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:35 -0400 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Prairial an 226 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 18 Jun 2018 15:48:33 +0200 Message-ID: <877emwurwu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) Dear thread lovers, The code below spawns a bunch of threads that look up imported bindings in a module. It usually enters an infinite loop and possibly eats all your memory (on Guile 2.2.3): --8<---------------cut here---------------start------------->8--- (use-modules (ice-9 threads) (ice-9 match) (srfi srfi-1)) (define bindings ;; Bindings exported by (guile). '()) (module-for-each (lambda (name var) (set! bindings (cons name bindings))) (resolve-module '(guile))) (define thread-count 8) (define bindings-per-thread (floor (/ (length bindings) thread-count))) (define iface (resolve-module '(ice-9 q))) (define threads (unfold (lambda (x) (>=3D x thread-count)) (lambda (n) (call-with-new-thread (lambda () (let loop ((bindings (take (drop bindings (* n bindings-per-thread)) bindings-per-thread))) (match bindings (() #t) ((head . tail) (module-variable iface head) (loop tail))))))) 1+ 0)) (for-each join-thread threads) (pk 'done thread-count) --8<---------------cut here---------------end--------------->8--- The issue lies in the =E2=80=9Cimport obarray=E2=80=9D, which is used as a = cache and is accessed in a non-thread-safe manner: --8<---------------cut here---------------start------------->8--- static inline SCM module_imported_variable (SCM module, SCM sym) { #define SCM_BOUND_THING_P scm_is_true register SCM var, imports; /* Search cached imported bindings. */ imports =3D SCM_MODULE_IMPORT_OBARRAY (module); var =3D scm_hashq_ref (imports, sym, SCM_UNDEFINED); if (SCM_BOUND_THING_P (var)) return var; [...] if (SCM_BOUND_THING_P (found_var)) { /* Save the lookup result for future reference. */ (void) scm_hashq_set_x (imports, sym, found_var); return found_var; } --8<---------------cut here---------------end--------------->8--- Possible solutions: 1. Add a field in the module record containing a mutex. Downside is that this breaks the ABI, though I=E2=80=99m not sure how much of a pr= oblem it is. 2. Make the import obarray a weak hash table since they are thread-safe in 2.2. This should be semantically equivalent to using a hash table provided interned symbols are not GC=E2=80=99d. Thoughts? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 18 09:53:16 2018 Received: (at control) by debbugs.gnu.org; 18 Jun 2018 13:53:16 +0000 Received: from localhost ([127.0.0.1]:54515 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUuap-00021m-Rw for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:53:16 -0400 Received: from eggs.gnu.org ([208.118.235.92]:42679) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUuao-00021Z-Ox for control@debbugs.gnu.org; Mon, 18 Jun 2018 09:53:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUuag-0006kT-A7 for control@debbugs.gnu.org; Mon, 18 Jun 2018 09:53:09 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38403) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUuag-0006kL-6X for control@debbugs.gnu.org; Mon, 18 Jun 2018 09:53:06 -0400 Received: from [193.50.110.191] (port=59486 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1fUuaf-0001zD-MI for control@debbugs.gnu.org; Mon, 18 Jun 2018 09:53:06 -0400 Date: Mon, 18 Jun 2018 15:53:03 +0200 Message-Id: <87602gurpc.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #31879 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) severity 31879 serious From unknown Sat Jun 14 19:33:49 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#31879: closed (Re: bug#31879: Caching in the "module import obarray" is not thread-safe) Message-ID: References: <871sd4ul22.fsf@gnu.org> <877emwurwu.fsf@gnu.org> X-Gnu-PR-Message: they-closed 31879 X-Gnu-PR-Package: guile Reply-To: 31879@debbugs.gnu.org Date: Mon, 18 Jun 2018 16:17:04 +0000 Content-Type: multipart/mixed; boundary="----------=_1529338624-28783-1" This is a multi-part message in MIME format... ------------=_1529338624-28783-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #31879: Caching in the "module import obarray" is not thread-safe which was filed against the guile package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 31879@debbugs.gnu.org. --=20 31879: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D31879 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1529338624-28783-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 31879-done) by debbugs.gnu.org; 18 Jun 2018 16:16:53 +0000 Received: from localhost ([127.0.0.1]:55216 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUwpp-0007TI-Ey for submit@debbugs.gnu.org; Mon, 18 Jun 2018 12:16:53 -0400 Received: from eggs.gnu.org ([208.118.235.92]:52311) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUwpm-0007T3-1G for 31879-done@debbugs.gnu.org; Mon, 18 Jun 2018 12:16:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUwpb-0003wJ-Tb for 31879-done@debbugs.gnu.org; Mon, 18 Jun 2018 12:16:44 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41609) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUwpb-0003wF-PG for 31879-done@debbugs.gnu.org; Mon, 18 Jun 2018 12:16:39 -0400 Received: from [193.50.110.191] (port=59886 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1fUwpb-0003qx-AI for 31879-done@debbugs.gnu.org; Mon, 18 Jun 2018 12:16:39 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: 31879-done@debbugs.gnu.org Subject: Re: bug#31879: Caching in the "module import obarray" is not thread-safe References: <877emwurwu.fsf@gnu.org> Date: Mon, 18 Jun 2018 18:16:37 +0200 In-Reply-To: <877emwurwu.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 18 Jun 2018 15:48:33 +0200") Message-ID: <871sd4ul22.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 31879-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > The issue lies in the =E2=80=9Cimport obarray=E2=80=9D, which is used as = a cache and is > accessed in a non-thread-safe manner: Fixed in commit 46bcbfa566de19a88e925bd0369e110cae5a6b03, following a suggestion by Andy. Ludo=E2=80=99. ------------=_1529338624-28783-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 18 Jun 2018 13:48:50 +0000 Received: from localhost ([127.0.0.1]:54504 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUuWY-0001ue-0Y for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:50 -0400 Received: from eggs.gnu.org ([208.118.235.92]:42011) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fUuWW-0001uS-Ub for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUuWQ-00054Q-SU for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:43 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:39829) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fUuWQ-00054M-Om for submit@debbugs.gnu.org; Mon, 18 Jun 2018 09:48:42 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:32977) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUuWM-0002K5-JX for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUuWJ-0004v8-Il for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:38 -0400 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38344) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUuWJ-0004ur-Ez for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:35 -0400 Received: from [193.50.110.191] (port=59408 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1fUuWJ-0001jF-23 for bug-guile@gnu.org; Mon, 18 Jun 2018 09:48:35 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: bug-guile@gnu.org Subject: Caching in the "module import obarray" is not thread-safe X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 30 Prairial an 226 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 18 Jun 2018 15:48:33 +0200 Message-ID: <877emwurwu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) Dear thread lovers, The code below spawns a bunch of threads that look up imported bindings in a module. It usually enters an infinite loop and possibly eats all your memory (on Guile 2.2.3): --8<---------------cut here---------------start------------->8--- (use-modules (ice-9 threads) (ice-9 match) (srfi srfi-1)) (define bindings ;; Bindings exported by (guile). '()) (module-for-each (lambda (name var) (set! bindings (cons name bindings))) (resolve-module '(guile))) (define thread-count 8) (define bindings-per-thread (floor (/ (length bindings) thread-count))) (define iface (resolve-module '(ice-9 q))) (define threads (unfold (lambda (x) (>=3D x thread-count)) (lambda (n) (call-with-new-thread (lambda () (let loop ((bindings (take (drop bindings (* n bindings-per-thread)) bindings-per-thread))) (match bindings (() #t) ((head . tail) (module-variable iface head) (loop tail))))))) 1+ 0)) (for-each join-thread threads) (pk 'done thread-count) --8<---------------cut here---------------end--------------->8--- The issue lies in the =E2=80=9Cimport obarray=E2=80=9D, which is used as a = cache and is accessed in a non-thread-safe manner: --8<---------------cut here---------------start------------->8--- static inline SCM module_imported_variable (SCM module, SCM sym) { #define SCM_BOUND_THING_P scm_is_true register SCM var, imports; /* Search cached imported bindings. */ imports =3D SCM_MODULE_IMPORT_OBARRAY (module); var =3D scm_hashq_ref (imports, sym, SCM_UNDEFINED); if (SCM_BOUND_THING_P (var)) return var; [...] if (SCM_BOUND_THING_P (found_var)) { /* Save the lookup result for future reference. */ (void) scm_hashq_set_x (imports, sym, found_var); return found_var; } --8<---------------cut here---------------end--------------->8--- Possible solutions: 1. Add a field in the module record containing a mutex. Downside is that this breaks the ABI, though I=E2=80=99m not sure how much of a pr= oblem it is. 2. Make the import obarray a weak hash table since they are thread-safe in 2.2. This should be semantically equivalent to using a hash table provided interned symbols are not GC=E2=80=99d. Thoughts? Ludo=E2=80=99. ------------=_1529338624-28783-1--