From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 30 16:35:24 2018 Received: (at submit) by debbugs.gnu.org; 30 Oct 2018 20:35:24 +0000 Received: from localhost ([127.0.0.1]:55942 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHajT-0007ZO-PV for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:35:23 -0400 Received: from eggs.gnu.org ([208.118.235.92]:50447) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHajS-0007ZC-I7 for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:35:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gHajM-0007te-NR for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:35:17 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:42670) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gHajL-0007tF-KZ for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:35:16 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60056) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gHajL-0003Ai-0c for guix-patches@gnu.org; Tue, 30 Oct 2018 16:35:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gHajH-0007q3-37 for guix-patches@gnu.org; Tue, 30 Oct 2018 16:35:14 -0400 Received: from mail.lassieur.org ([83.152.10.219]:59404) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gHajG-0007lQ-MA for guix-patches@gnu.org; Tue, 30 Oct 2018 16:35:11 -0400 Received: from rodion (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id 08fe3b90 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO) for ; Tue, 30 Oct 2018 20:33:47 +0000 (UTC) User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: guix-patches@gnu.org Subject: Cuirass: Use a SQLite in single-thread mode Date: Tue, 30 Oct 2018 21:35:06 +0100 Message-ID: <87a7mvqikl.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.1 (-----) Hi, These patches are supposed to slightly improve Cuirass' performances, because it doesn't use the multi-threading features. Cl=C3=A9ment From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 30 16:47:41 2018 Received: (at 33210) by debbugs.gnu.org; 30 Oct 2018 20:47:41 +0000 Received: from localhost ([127.0.0.1]:55948 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHavM-0007tu-Sj for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:47:41 -0400 Received: from mail.lassieur.org ([83.152.10.219]:48280) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHavL-0007tl-Dy for 33210@debbugs.gnu.org; Tue, 30 Oct 2018 16:47:40 -0400 Received: from localhost.localdomain (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id b3644b6a (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for <33210@debbugs.gnu.org>; Tue, 30 Oct 2018 20:46:17 +0000 (UTC) From: =?UTF-8?q?Cl=C3=A9ment=20Lassieur?= To: 33210@debbugs.gnu.org Subject: [PATCH 1/3] gnu: Add sqlite-with-single-thread. Date: Tue, 30 Oct 2018 21:47:24 +0100 Message-Id: <20181030204726.13354-1-clement@lassieur.org> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * gnu/packages/databases.scm (sqlite-with-single-thread): New variable. --- gnu/packages/databases.scm | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm index 87fb170e5..24914cd87 100644 --- a/gnu/packages/databases.scm +++ b/gnu/packages/databases.scm @@ -32,6 +32,7 @@ ;;; Copyright © 2017 Kristofer Buffington ;;; Copyright © 2018 Amirouche Boubekki ;;; Copyright © 2018 Joshua Sierles, Nextjournal +;;; Copyright © 2018 Clément Lassieur ;;; ;;; This file is part of GNU Guix. ;;; @@ -1223,6 +1224,15 @@ is in the public domain.") "-DSQLITE_ENABLE_DBSTAT_VTAB " "-DSQLITE_ENABLE_COLUMN_METADATA"))))))) +;; This is used by Cuirass. +(define-public sqlite-with-single-thread + (package (inherit sqlite) + (name "sqlite-with-single-thread") + (arguments + (substitute-keyword-arguments (package-arguments sqlite) + ((#:configure-flags flags) + `(cons "--disable-threadsafe" ,flags)))))) + (define-public tdb (package (name "tdb") -- 2.19.1 From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 30 16:47:44 2018 Received: (at 33210) by debbugs.gnu.org; 30 Oct 2018 20:47:44 +0000 Received: from localhost ([127.0.0.1]:55951 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHavQ-0007uF-5S for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:47:44 -0400 Received: from mail.lassieur.org ([83.152.10.219]:48280) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHavM-0007tl-DP for 33210@debbugs.gnu.org; Tue, 30 Oct 2018 16:47:41 -0400 Received: from localhost.localdomain (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id cdb2e5b7 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for <33210@debbugs.gnu.org>; Tue, 30 Oct 2018 20:46:17 +0000 (UTC) From: =?UTF-8?q?Cl=C3=A9ment=20Lassieur?= To: 33210@debbugs.gnu.org Subject: [PATCH 2/3] gnu: Add guile-sqlite3-with-single-thread. Date: Tue, 30 Oct 2018 21:47:25 +0100 Message-Id: <20181030204726.13354-2-clement@lassieur.org> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181030204726.13354-1-clement@lassieur.org> References: <20181030204726.13354-1-clement@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * gnu/packages/guile.scm (guile-sqlite3-with-single-thread): New variable. --- gnu/packages/guile.scm | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm index 9e3300337..4f9ddf913 100644 --- a/gnu/packages/guile.scm +++ b/gnu/packages/guile.scm @@ -20,6 +20,7 @@ ;;; Copyright © 2018 Arun Isaac ;;; Copyright © 2018 Pierre-Antoine Rouby ;;; Copyright © 2018 Eric Bavier +;;; Copyright © 2018 Clément Lassieur ;;; ;;; This file is part of GNU Guix. ;;; @@ -71,6 +72,7 @@ #:use-module (gnu packages xdisorg) #:use-module (gnu packages xorg) #:use-module (gnu packages networking) + #:use-module ((guix build utils) #:select (alist-replace)) #:use-module (guix packages) #:use-module (guix download) #:use-module (guix git-download) @@ -1166,6 +1168,13 @@ Guile's foreign function interface.") "This package provides Guile bindings to the SQLite database system.") (license license:gpl3+))) +(define-public guile-sqlite3-with-single-thread + (package + (inherit guile-sqlite3) + (name "guile-sqlite3-with-single-thread") + (inputs (alist-replace "sqlite" (list sqlite-with-single-thread) + (package-inputs guile-sqlite3))))) + (define-public guile2.0-sqlite3 (package-for-guile-2.0 guile-sqlite3)) -- 2.19.1 From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 30 16:47:44 2018 Received: (at 33210) by debbugs.gnu.org; 30 Oct 2018 20:47:44 +0000 Received: from localhost ([127.0.0.1]:55953 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHavQ-0007uI-HT for submit@debbugs.gnu.org; Tue, 30 Oct 2018 16:47:44 -0400 Received: from mail.lassieur.org ([83.152.10.219]:48280) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHavN-0007tl-GK for 33210@debbugs.gnu.org; Tue, 30 Oct 2018 16:47:41 -0400 Received: from localhost.localdomain (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id 777aa4e7 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for <33210@debbugs.gnu.org>; Tue, 30 Oct 2018 20:46:17 +0000 (UTC) From: =?UTF-8?q?Cl=C3=A9ment=20Lassieur?= To: 33210@debbugs.gnu.org Subject: [PATCH 3/3] gnu: cuirass: Use SQLite in single-thread mode. Date: Tue, 30 Oct 2018 21:47:26 +0100 Message-Id: <20181030204726.13354-3-clement@lassieur.org> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181030204726.13354-1-clement@lassieur.org> References: <20181030204726.13354-1-clement@lassieur.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * gnu/packages/ci.scm (cuirass)[inputs]: Replace GUILE-SQLITE3 with GUILE-SQLITE3-WITH-SINGLE-THREAD. --- gnu/packages/ci.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gnu/packages/ci.scm b/gnu/packages/ci.scm index 1cac8b9fb..126948109 100644 --- a/gnu/packages/ci.scm +++ b/gnu/packages/ci.scm @@ -261,7 +261,7 @@ their dependencies.") ("guile-fibers" ,guile-fibers) ("guile-gcrypt" ,guile-gcrypt) ("guile-json" ,guile-json) - ("guile-sqlite3" ,guile-sqlite3) + ("guile-sqlite3" ,guile-sqlite3-with-single-thread) ("guile-git" ,guile-git) ;; FIXME: this is propagated by "guile-git", but it needs to be among ;; the inputs to add it to GUILE_LOAD_PATH. -- 2.19.1 From debbugs-submit-bounces@debbugs.gnu.org Sun Nov 04 18:08:37 2018 Received: (at 33210) by debbugs.gnu.org; 4 Nov 2018 23:08:37 +0000 Received: from localhost ([127.0.0.1]:34418 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJRVV-00036k-3Q for submit@debbugs.gnu.org; Sun, 04 Nov 2018 18:08:37 -0500 Received: from eggs.gnu.org ([208.118.235.92]:38643) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJRVT-00036W-Gc for 33210@debbugs.gnu.org; Sun, 04 Nov 2018 18:08:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gJRVL-0006Bl-Br for 33210@debbugs.gnu.org; Sun, 04 Nov 2018 18:08:29 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41412) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gJRVI-00069A-HB; Sun, 04 Nov 2018 18:08:25 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=34368 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1gJRVI-0000Gv-63; Sun, 04 Nov 2018 18:08:24 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: =?utf-8?Q?Cl=C3=A9ment?= Lassieur Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode References: <87a7mvqikl.fsf@lassieur.org> Date: Mon, 05 Nov 2018 00:08:23 +0100 In-Reply-To: <87a7mvqikl.fsf@lassieur.org> (=?utf-8?Q?=22Cl=C3=A9ment?= Lassieur"'s message of "Tue, 30 Oct 2018 21:35:06 +0100") Message-ID: <871s80o2zc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) Hello, Cl=C3=A9ment Lassieur skribis: > These patches are supposed to slightly improve Cuirass' performances, > because it doesn't use the multi-threading features. Did you notice a measurable difference? We could do it, but IMO that should be a last resort because I=E2=80=99d ex= pect it to be a micro-optimization. WDYT? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Nov 05 03:03:02 2018 Received: (at 33210) by debbugs.gnu.org; 5 Nov 2018 08:03:02 +0000 Received: from localhost ([127.0.0.1]:34549 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJZqg-0007Rx-8O for submit@debbugs.gnu.org; Mon, 05 Nov 2018 03:03:02 -0500 Received: from mail.lassieur.org ([83.152.10.219]:41872) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJZqe-0007Ri-Q2 for 33210@debbugs.gnu.org; Mon, 05 Nov 2018 03:03:01 -0500 Received: from rodion (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id ab38c014 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Mon, 5 Nov 2018 08:00:40 +0000 (UTC) References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode In-reply-to: <871s80o2zc.fsf@gnu.org> Date: Mon, 05 Nov 2018 09:02:57 +0100 Message-ID: <87ftwgq7da.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Hello, > > Cl=C3=A9ment Lassieur skribis: > >> These patches are supposed to slightly improve Cuirass' performances, >> because it doesn't use the multi-threading features. > > Did you notice a measurable difference? I haven't done any measurement yet, but according to the SQLite documentation: Setting -DSQLITE_THREADSAFE=3D0 causes all of the mutex and thread-safety logic in SQLite to be omitted. This is the single compile-time option that makes the most difference in optimizing the performance of SQLite. So even if the optimization is small, it's the option that has the biggest impact on performance. > We could do it, but IMO that should be a last resort because I=E2=80=99d = expect > it to be a micro-optimization. Lots of micro-optimizations lead to an overall faster application ;-). And this one doesn't make the code more complicated. To me it's just a bonus. [1]: https://www.sqlite.org/compile.html From debbugs-submit-bounces@debbugs.gnu.org Mon Nov 05 19:12:02 2018 Received: (at 33210) by debbugs.gnu.org; 6 Nov 2018 00:12:02 +0000 Received: from localhost ([127.0.0.1]:35842 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJoyO-0001P4-IT for submit@debbugs.gnu.org; Mon, 05 Nov 2018 19:12:02 -0500 Received: from dd26836.kasserver.com ([85.13.145.193]:35358) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJoyM-0001Ow-GW for 33210@debbugs.gnu.org; Mon, 05 Nov 2018 19:11:59 -0500 Received: from localhost (178.113.175.117.wireless.dyn.drei.com [178.113.175.117]) by dd26836.kasserver.com (Postfix) with ESMTPSA id 97D3F3360156; Tue, 6 Nov 2018 01:11:56 +0100 (CET) Date: Tue, 6 Nov 2018 01:11:54 +0100 From: Danny Milosavljevic To: =?ISO-8859-1?Q?Cl=E9ment?= Lassieur Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode Message-ID: <20181106011154.3f235763@scratchpost.org> In-Reply-To: <87ftwgq7da.fsf@lassieur.org> References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/u0q.3rbDce6kZRVK/C3r2wj"; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org, Ludovic =?ISO-8859-1?Q?Court=E8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --Sig_/u0q.3rbDce6kZRVK/C3r2wj Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Cl=C3=A9ment, > I haven't done any measurement yet, but according to the SQLite > documentation: >=20 > Setting -DSQLITE_THREADSAFE=3D0 causes all of the mutex and > thread-safety logic in SQLite to be omitted. This is the single > compile-time option that makes the most difference in optimizing the > performance of SQLite. >=20 > So even if the optimization is small, it's the option that has the > biggest impact on performance. >=20 > > We could do it, but IMO that should be a last resort because I=E2=80=99= d expect > > it to be a micro-optimization. =20 >=20 > Lots of micro-optimizations lead to an overall faster application ;-). > And this one doesn't make the code more complicated. To me it's just a > bonus. Keep in mind that if we want consistent views via the web interface, the cuirass evaluator has to use its own connection independent of the web interface (so that the web interface doesn't see half-finished stuff). If that's still possible after that then fine. Right now, as I mentioned before, it can happen that you request a certain filter when requesting something from the web and the result will actually contain data that does not match the filter. What happened is that the data in the transaction got changed before we returned it but after the first query ran. This is not supposed to happen in relational database systems. The reason why it happens here is because we use the same transaction (session) for both the updates done by the evaluator and the queries done by the web interface. It would be much better if the queries by the web interface had their own transaction. It was fine before but in some refactoring, "evaluate" ceased to be an external program and I didn't notice what happened to it. --Sig_/u0q.3rbDce6kZRVK/C3r2wj Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvg3EoACgkQ5xo1VCww uqVy8AgAjYGaTge0p2rpUl6eK31W3VZQ9C5qRSnFyDrv07W3q4X/5TGNrY3GKKKQ uwdq2eODz2Ro4HHI/4oclNPIvwhssECdQHlDEYPymwcK41AtBixiE+IpzaPgyya7 hY394Q/h45SXK4+ZnnKVzjhpP+3AShFP1wvvCifZCW4JilaAOSpIaqlbLD9YymwI C+OhM6hZFBaI8AjnzG9fxc2ujBozFVW8i94Y7vykTJgTc4ua//4SgY1jogYgsEQf elUwAvTSA7C4ClSoIWZ96JS2gmCssufR9e585yxZlf3prTXrt38K6cPPGVmg8dHC TrHidrTNtBL+FD1PSZOyLoSSxc09rQ== =CkT6 -----END PGP SIGNATURE----- --Sig_/u0q.3rbDce6kZRVK/C3r2wj-- From debbugs-submit-bounces@debbugs.gnu.org Mon Nov 05 19:50:15 2018 Received: (at 33210) by debbugs.gnu.org; 6 Nov 2018 00:50:15 +0000 Received: from localhost ([127.0.0.1]:35855 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJpZO-0002Kv-Tz for submit@debbugs.gnu.org; Mon, 05 Nov 2018 19:50:15 -0500 Received: from mail.lassieur.org ([83.152.10.219]:44690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJpZN-0002Kn-PY for 33210@debbugs.gnu.org; Mon, 05 Nov 2018 19:50:14 -0500 Received: from rodion (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id 801866ae (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Tue, 6 Nov 2018 00:47:45 +0000 (UTC) References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Danny Milosavljevic Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode In-reply-to: <20181106011154.3f235763@scratchpost.org> Date: Tue, 06 Nov 2018 01:50:11 +0100 Message-ID: <87efbzqbb0.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org, Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Danny, Danny Milosavljevic writes: > Keep in mind that if we want consistent views via the web interface, > the cuirass evaluator has to use its own connection independent of the > web interface (so that the web interface doesn't see half-finished stuff). > If that's still possible after that then fine. > > Right now, as I mentioned before, it can happen that you request a certain > filter when requesting something from the web and the result will actually > contain data that does not match the filter. What happened is that the > data in the transaction got changed before we returned it but after the > first query ran. > > This is not supposed to happen in relational database systems. The reason > why it happens here is because we use the same transaction (session) for > both the updates done by the evaluator and the queries done by the web > interface. It would be much better if the queries by the web interface > had their own transaction. It was fine before but in some refactoring, > "evaluate" ceased to be an external program and I didn't notice what > happened to it. Yes, I know, but I remember that I told you[1] that there was an easy (one line) fix for this. :-) Do you want to send a patch? [1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32234#57 From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 06 06:20:44 2018 Received: (at 33210) by debbugs.gnu.org; 6 Nov 2018 11:20:44 +0000 Received: from localhost ([127.0.0.1]:36068 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJzPY-0000pO-8i for submit@debbugs.gnu.org; Tue, 06 Nov 2018 06:20:44 -0500 Received: from dd26836.kasserver.com ([85.13.145.193]:36500) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gJzPW-0000pG-DJ for 33210@debbugs.gnu.org; Tue, 06 Nov 2018 06:20:42 -0500 Received: from localhost (178.113.175.117.wireless.dyn.drei.com [178.113.175.117]) by dd26836.kasserver.com (Postfix) with ESMTPSA id D9145336024E; Tue, 6 Nov 2018 12:20:40 +0100 (CET) Date: Tue, 6 Nov 2018 12:20:36 +0100 From: Danny Milosavljevic To: =?ISO-8859-1?Q?Cl=E9ment?= Lassieur Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode Message-ID: <20181106122036.25bad548@scratchpost.org> In-Reply-To: <87efbzqbb0.fsf@lassieur.org> References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> <87efbzqbb0.fsf@lassieur.org> X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/Z4td89Gr1XPGdvVZtoaQWFI"; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org, Ludovic =?ISO-8859-1?Q?Court=E8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --Sig_/Z4td89Gr1XPGdvVZtoaQWFI Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Cl=C3=A9ment, On Tue, 06 Nov 2018 01:50:11 +0100 Cl=C3=A9ment Lassieur wrote: > > This is not supposed to happen in relational database systems. The rea= son > > why it happens here is because we use the same transaction (session) for > > both the updates done by the evaluator and the queries done by the web > > interface. It would be much better if the queries by the web interface > > had their own transaction. It was fine before but in some refactoring, > > "evaluate" ceased to be an external program and I didn't notice what > > happened to it. =20 >=20 > Yes, I know, but I remember that I told you[1] that there was an easy > (one line) fix for this. :-) That would really only be a workaround (arguably still better than what we have now - which is basically you ask for a banana and I give you an apple - what if other procedures in cuirass assume it's a banana? :) ). We would be doing the work that SQLite would have done, but we'd do it in a= bad way (by blocking everyone else). It's not really useful to undo all the progress relational databases have m= ade. If we had multiple transactions in progress touching the same row, SQLite w= ould use MVCC to hold diverging copies of the same row at the same time, blocking nobody. --Sig_/Z4td89Gr1XPGdvVZtoaQWFI Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvheQQACgkQ5xo1VCww uqU96QgApiUGURnK3NCZm+0DtLotX3+N9jEcZcxN00qftsYSRUdbF/niNNVpGlNA Xeb1dTeYGJxoaGH69YdeJJKe7rdEBlNg/PluKoDMxoVrXeKcDUYsRNt8aT7BxCz0 HWWSZS5izcU2HoUy/JPNUnoe8yNnoIl4x6cJq5DQI4sw5PFfqtXtoxL1ZnYNsm8y EbzBObbTTmHU4VPHDTfowhtryGVUhlGgz3JoA9y0XjCzZ1JXgD0v7gdB8WrPndla xgaCie1WerUlhydgWGz4k/74j40AxZ/VNzABoldkwc1cCSiM5E61V3z3UMVst/bc GX2+RAEA+a1MGmLSVIRd0CBY9aKgpw== =Cd91 -----END PGP SIGNATURE----- --Sig_/Z4td89Gr1XPGdvVZtoaQWFI-- From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 06 09:07:46 2018 Received: (at 33210) by debbugs.gnu.org; 6 Nov 2018 14:07:46 +0000 Received: from localhost ([127.0.0.1]:36161 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK21B-00077T-PJ for submit@debbugs.gnu.org; Tue, 06 Nov 2018 09:07:45 -0500 Received: from mail.lassieur.org ([83.152.10.219]:46958) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK218-00077F-Ru for 33210@debbugs.gnu.org; Tue, 06 Nov 2018 09:07:43 -0500 Received: from newt (smtp.parrot.biz [62.23.167.188]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id f36eea66 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Tue, 6 Nov 2018 14:05:08 +0000 (UTC) References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> <87efbzqbb0.fsf@lassieur.org> <20181106122036.25bad548@scratchpost.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Danny Milosavljevic Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode In-reply-to: <20181106122036.25bad548@scratchpost.org> X-TUID: yHcfq6aHriOZ Date: Tue, 06 Nov 2018 15:07:40 +0100 Message-ID: <875zxa5mfn.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hey Danny, Danny Milosavljevic writes: > Hi Cl=C3=A9ment, > > On Tue, 06 Nov 2018 01:50:11 +0100 > Cl=C3=A9ment Lassieur wrote: > >> > This is not supposed to happen in relational database systems. The re= ason >> > why it happens here is because we use the same transaction (session) f= or >> > both the updates done by the evaluator and the queries done by the web >> > interface. It would be much better if the queries by the web interface >> > had their own transaction. It was fine before but in some refactoring, >> > "evaluate" ceased to be an external program and I didn't notice what >> > happened to it.=20=20 >>=20 >> Yes, I know, but I remember that I told you[1] that there was an easy >> (one line) fix for this. :-) > > That would really only be a workaround (arguably still better than what we > have now - which is basically you ask for a banana and I give you an apple > - what if other procedures in cuirass assume it's a banana? :) ). > > We would be doing the work that SQLite would have done, but we'd do it in= a bad > way (by blocking everyone else). > > It's not really useful to undo all the progress relational databases have= made. > > If we had multiple transactions in progress touching the same row, SQLite= would > use MVCC to hold diverging copies of the same row at the same time, block= ing > nobody. It blocks everyone indeed, because we take care of the scheduling in a rather basic way. But if I understand correctly, the overall spent time is more or less the same: only the order of the requests differs. There is an essential difference however: if we take care of the scheduling, we won't have SQLITE_BUSY blocking the Fibers scheduler all the time. And blocking the Fibers scheduler has an impact on all other possibly unrelated Fibers clients. When guile-sqlite3 handles SQLITE_BUSY correctly, I'll be happy to switch back to a multi-threading SQLite. While waiting for it, I believe our users want a fast Cuirass, and I'd like the time spent in the Fibers scheduler to be balanced by removing the SQLite now useless mutexes. Does that make sense? Cl=C3=A9ment From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 06 09:40:22 2018 Received: (at 33210) by debbugs.gnu.org; 6 Nov 2018 14:40:22 +0000 Received: from localhost ([127.0.0.1]:36201 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK2Wk-0001dL-6a for submit@debbugs.gnu.org; Tue, 06 Nov 2018 09:40:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:46733) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK2Wi-0001d8-JY for 33210@debbugs.gnu.org; Tue, 06 Nov 2018 09:40:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gK2Wc-0001td-DR for 33210@debbugs.gnu.org; Tue, 06 Nov 2018 09:40:15 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:54443) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gK2Wc-0001tK-8l; Tue, 06 Nov 2018 09:40:14 -0500 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=33356 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1gK2Wb-0006K9-V9; Tue, 06 Nov 2018 09:40:14 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: =?utf-8?Q?Cl=C3=A9ment?= Lassieur Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 16 Brumaire an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 06 Nov 2018 15:40:11 +0100 In-Reply-To: <87ftwgq7da.fsf@lassieur.org> (=?utf-8?Q?=22Cl=C3=A9ment?= Lassieur"'s message of "Mon, 05 Nov 2018 09:02:57 +0100") Message-ID: <87ftwez2us.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) Hi Cl=C3=A9ment, Cl=C3=A9ment Lassieur skribis: > Ludovic Court=C3=A8s writes: > >> Hello, >> >> Cl=C3=A9ment Lassieur skribis: >> >>> These patches are supposed to slightly improve Cuirass' performances, >>> because it doesn't use the multi-threading features. >> >> Did you notice a measurable difference? > > I haven't done any measurement yet, but according to the SQLite > documentation: > > Setting -DSQLITE_THREADSAFE=3D0 causes all of the mutex and > thread-safety logic in SQLite to be omitted. This is the single > compile-time option that makes the most difference in optimizing the > performance of SQLite. > > So even if the optimization is small, it's the option that has the > biggest impact on performance. > >> We could do it, but IMO that should be a last resort because I=E2=80=99d= expect >> it to be a micro-optimization. > > Lots of micro-optimizations lead to an overall faster application ;-). > And this one doesn't make the code more complicated. To me it's just a > bonus. I agree it doesn=E2=80=99t complicate the code; still, that=E2=80=99s a cou= ple of additional package variants to deal with, for hardly measurable benefits I suspect. I think we should focus on higher-level optimizations at this development stage of Cuirass. For instance I have been meaning to patch it so that it doesn=E2=80=99t have to process all the build logs, since it doesn=E2=80=99t do anything with those logs and processing them involves to= ns of syscalls and string processing and introduces latency in fiber scheduling. This is a simple change that could have a more visible impact I believe. Hopefully I=E2=80=99ll get there real soon=E2=80=A6 WDYT? Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 06 15:10:59 2018 Received: (at 33210) by debbugs.gnu.org; 6 Nov 2018 20:10:59 +0000 Received: from localhost ([127.0.0.1]:37838 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK7gg-0007mg-Vj for submit@debbugs.gnu.org; Tue, 06 Nov 2018 15:10:59 -0500 Received: from dd26836.kasserver.com ([85.13.145.193]:59990) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gK7gf-0007mY-NV for 33210@debbugs.gnu.org; Tue, 06 Nov 2018 15:10:58 -0500 Received: from localhost (178.113.175.117.wireless.dyn.drei.com [178.113.175.117]) by dd26836.kasserver.com (Postfix) with ESMTPSA id 29934336024E; Tue, 6 Nov 2018 21:10:56 +0100 (CET) Date: Tue, 6 Nov 2018 21:10:49 +0100 From: Danny Milosavljevic To: =?ISO-8859-1?Q?Cl=E9ment?= Lassieur Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode Message-ID: <20181106211049.1469331b@scratchpost.org> In-Reply-To: <875zxa5mfn.fsf@lassieur.org> References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> <87efbzqbb0.fsf@lassieur.org> <20181106122036.25bad548@scratchpost.org> <875zxa5mfn.fsf@lassieur.org> X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/wcdjKIw1cVlVAXsyYdXyGHn"; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --Sig_/wcdjKIw1cVlVAXsyYdXyGHn Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Cl=C3=A9ment, > rather basic way. But if I understand correctly, the overall spent time > is more or less the same: only the order of the requests differs. Yeah, right now users can query something using the web interface while a build is updating (or running) at the cost of the returned data being potentially very strange. The "one-line fix" would make it worse in that users cannot query while a build is running, making them wait until the build is done (approx. 30 min) before the query succeeds. The upside is that the returned data is consistent at all times. This is how DBMSes did it in the 90s, too. What I'd like to eventually have is the proper fix where users can query while a build is running, *and* the build doesn't have to wait either. This works just fine using two transactions with WAL mode of sqlite, which means it uses MVCC in order to keep both "world views", one for the querier and one for the builder (easily extended to an arbitrary number of queriers and builders at once by just having more transactions) while they are both using "the world". > is an essential difference however: if we take care of the scheduling, > we won't have SQLITE_BUSY blocking the Fibers scheduler all the time. > And blocking the Fibers scheduler has an impact on all other possibly > unrelated Fibers clients. Right. I just wanted to make sure we understand the possible implications = here. In the end I'm not sure we even need multithreading even for my scenario - maybe (probably) just having an extra sqlite_open would be enough, threads or not. On the other hand there are shared caches etc and this change here could cause some very tricky problems then. I have to say I liked the external evaluator much more since then all this complexity would be contained in the external program and it would just magically work without special-casing any of this stuff. > When guile-sqlite3 handles SQLITE_BUSY > correctly, I'll be happy to switch back to a multi-threading SQLite. > While waiting for it, I believe our users want a fast Cuirass, and I'd > like the time spent in the Fibers scheduler to be balanced by removing > the SQLite now useless mutexes. That makes sense. It's difficult for guile-sqlite3 to handle SQLITE_BUSY correctly since sqlite also uses SQLITE_BUSY to indicate errors that you are supposed to fail on. In the non-presence of a busy handler, it's not possible to distinguish whether the SQLITE_BUSY was of the "please retry" kind or of the "don't you retry" kind. It would mean that guile-sqlite3 would have to have its own flag that indicates whether the busy handler was called, and check this one. Resetting this flag would also have to be potentially thread-safe (for other users of guile-sqlite3). That's always assuming that sqlite3 undos whatever it was trying to do before returning SQLITE_BUSY so it actually makes sense to retry the call later. So something like this: guile_sqlite_handle_busy(...) { guile_struct->busy_handler_called =3D true; return 0; // fail } guile_sqlite_open { int rc =3D native_sqlite_open(...); native_sqlite_set_busy_handler(..., guile_sqlite_handle_busy); // FIXME: check for errors here and fail on error guile_struct->busy_handler_called =3D false; } guile_sqlite_method { int rc, busy_handler_called; do { rc =3D native_sqlite_method(...); } while (rc =3D=3D SQLITE_BUSY && (busy_handler_called =3D test-and-reset= (guile_struct->busy_handler_called), yield)); return rc; } Hmmmmmmmm. I think that can be done. Notes for myself: pager.c busyHandler btreeInvokeBusyHandler sqlite3BtreeBeginTrans sqlite3PagerSetBusyhandler SQLITE_FCNTL_BUSYHANDLER=20 --Sig_/wcdjKIw1cVlVAXsyYdXyGHn Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvh9UkACgkQ5xo1VCww uqUe9Af/YUIZmFoDif6PysDEoFvsc7ZkyOCSAfoboENtcif3JHTzIDkuf0zXwZUP IsPQR5SHzj4jJXjfhex1QPDAK3YB16GmCwkVPfXAfj694fbJz560MGt6kzajMyv2 clvUZRO9Xdb9bv/j4+V39U52Xy7UQeX8pGLIp8LCkZQYpcU4Ho/J0f5AVe2mABhR 1iPLTWgK15B4cvu2ztY7mV1lGI06uF8TCAFcl9yxWUO3Bc3NBxJ3JgXEk/5B60ml Grd6dRaoe8fpVx8K50KjAMO5qIGY2HITNpPfK7b57yPVMmsJ5wzQ+ApueAluviNa EaLiBac6L/0TiF+srMSkUofqpLytTg== =Gwg+ -----END PGP SIGNATURE----- --Sig_/wcdjKIw1cVlVAXsyYdXyGHn-- From debbugs-submit-bounces@debbugs.gnu.org Wed Nov 07 06:38:25 2018 Received: (at 33210) by debbugs.gnu.org; 7 Nov 2018 11:38:26 +0000 Received: from localhost ([127.0.0.1]:38200 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gKMAD-000755-Kp for submit@debbugs.gnu.org; Wed, 07 Nov 2018 06:38:25 -0500 Received: from mail.lassieur.org ([83.152.10.219]:50578) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gKMAA-00074u-To for 33210@debbugs.gnu.org; Wed, 07 Nov 2018 06:38:23 -0500 Received: from newt (smtp.parrot.biz [62.23.167.188]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id f5f643c7 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Wed, 7 Nov 2018 11:35:38 +0000 (UTC) References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> <87efbzqbb0.fsf@lassieur.org> <20181106122036.25bad548@scratchpost.org> <875zxa5mfn.fsf@lassieur.org> <20181106211049.1469331b@scratchpost.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Danny Milosavljevic Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode In-reply-to: <20181106211049.1469331b@scratchpost.org> Date: Wed, 07 Nov 2018 12:38:20 +0100 Message-ID: <8736sd5d8z.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Danny, Danny Milosavljevic writes: > Yeah, right now users can query something using the web interface while > a build is updating (or running) at the cost of the returned data being > potentially very strange. This is quite unlikely. > The "one-line fix" would make it worse in that users cannot query while > a build is running, making them wait until the build is done (approx. > 30 min) before the query succeeds. The upside is that the returned > data is consistent at all times. This is how DBMSes did it in the 90s, > too. No, there are no SQL requests during the build. There are some before the build, and some after the build. But they are all short. > What I'd like to eventually have is the proper fix where users can query > while a build is running, *and* the build doesn't have to wait either. > This works just fine using two transactions with WAL mode of sqlite, > which means it uses MVCC in order to keep both "world views", one for the > querier and one for the builder (easily extended to an arbitrary > number of queriers and builders at once by just having more transactions) > while they are both using "the world". It's already the case, because all the queries are very short. >> is an essential difference however: if we take care of the scheduling, >> we won't have SQLITE_BUSY blocking the Fibers scheduler all the time. >> And blocking the Fibers scheduler has an impact on all other possibly >> unrelated Fibers clients. > > Right. I just wanted to make sure we understand the possible implication= s here. > > In the end I'm not sure we even need multithreading even for my scenario - > maybe (probably) just having an extra sqlite_open would be enough, threads > or not. On the other hand there are shared caches etc and this change he= re > could cause some very tricky problems then. I don't understand this. Could you explain why we would need an extra sqlite_open, what change are you talking about, and why there could be tricky problems with shared caches? > I have to say I liked the external evaluator much more since then all > this complexity would be contained in the external program and it would > just magically work without special-casing any of this stuff. The evaluator is still external, I'm not sure what you are talking about. >> When guile-sqlite3 handles SQLITE_BUSY >> correctly, I'll be happy to switch back to a multi-threading SQLite. >> While waiting for it, I believe our users want a fast Cuirass, and I'd >> like the time spent in the Fibers scheduler to be balanced by removing >> the SQLite now useless mutexes. > > That makes sense. > > It's difficult for guile-sqlite3 to handle SQLITE_BUSY correctly since > sqlite also uses SQLITE_BUSY to indicate errors that you are supposed to > fail on. [...] > Hmmmmmmmm. I think that can be done. Cool! Cheers, Cl=C3=A9ment From debbugs-submit-bounces@debbugs.gnu.org Wed Nov 07 18:59:48 2018 Received: (at 33210) by debbugs.gnu.org; 7 Nov 2018 23:59:48 +0000 Received: from localhost ([127.0.0.1]:40121 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gKXjf-0002HA-PZ for submit@debbugs.gnu.org; Wed, 07 Nov 2018 18:59:47 -0500 Received: from dd26836.kasserver.com ([85.13.145.193]:53648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gKXje-0002H3-AS for 33210@debbugs.gnu.org; Wed, 07 Nov 2018 18:59:46 -0500 Received: from localhost (178.112.138.94.wireless.dyn.drei.com [178.112.138.94]) by dd26836.kasserver.com (Postfix) with ESMTPSA id 48238336142B; Thu, 8 Nov 2018 00:59:45 +0100 (CET) Date: Thu, 8 Nov 2018 00:59:41 +0100 From: Danny Milosavljevic To: =?ISO-8859-1?Q?Cl=E9ment?= Lassieur Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode Message-ID: <20181108005941.519c464e@scratchpost.org> In-Reply-To: <8736sd5d8z.fsf@lassieur.org> References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> <87efbzqbb0.fsf@lassieur.org> <20181106122036.25bad548@scratchpost.org> <875zxa5mfn.fsf@lassieur.org> <20181106211049.1469331b@scratchpost.org> <8736sd5d8z.fsf@lassieur.org> X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-unknown-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; boundary="Sig_/8EbDUZu_Pc=BJmIvJg7IYfq"; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) --Sig_/8EbDUZu_Pc=BJmIvJg7IYfq Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Cl=C3=A9ment, > > Yeah, right now users can query something using the web interface while > > a build is updating (or running) at the cost of the returned data being > > potentially very strange. =20 >=20 > This is quite unlikely. When testing the cuirass status frontend it happened regularily without me trying to make it happen - took quite some time to find the cause, too. > No, there are no SQL requests during the build. There are some before > the build, and some after the build. But they are all short. Yes, if there is one transaction right before starting the build and another transaction at the end of the build, then it's much better. > > I have to say I liked the external evaluator much more since then all > > this complexity would be contained in the external program and it would > > just magically work without special-casing any of this stuff. =20 >=20 > The evaluator is still external, I'm not sure what you are talking > about. Hmm, I'll read through the source for a bit. I was of the impression that now the cuirass main process did the updating of the build status rather than the evaluator. > > It's difficult for guile-sqlite3 to handle SQLITE_BUSY correctly since > > sqlite also uses SQLITE_BUSY to indicate errors that you are supposed to > > fail on. =20 >=20 > [...] >=20 > > Hmmmmmmmm. I think that can be done. =20 I've tried it and it works well enough, although some of the sqlite documentation makes it sound like one cannot *just* retry some of the calls (for example: sqlite3_step). It's one of the disadantages of fibers that every C library has to have special code in it to support it somehow (if at all) - it means that it has to be written in a way to make all calls non-blocking. --Sig_/8EbDUZu_Pc=BJmIvJg7IYfq Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEds7GsXJ0tGXALbPZ5xo1VCwwuqUFAlvjfG0ACgkQ5xo1VCww uqURCgf/SZxWnXDv6HnO4YcOkhxyvJN5Lw7Oz9vffaD/CiByOJR8y+ELGdfpycey IdhTaTydcncOyjV4cKvJEdQN/msoLJ/sUHgc7/YqFje8LbZUbHPgM3Z3EJ8U5Dw5 FHoq772ahaS4NmL450Vgh7p0oyDsmW9vuank5qBR8BxlQfJMcNtg0WDs1kY8pWjZ DbCvhJx7LR2yxO06kNEeGno4KHJcKit/vFJueroEVEzcLSapjnF1U4Xcpyzg5V2Q ui/GrUILBHAmdmdwG4L1d036zhJqVgKcDYiup9VS8byYtf2XoWsYvN9M4BuZzj3Y wqqCys5S/XxpXp/kAoSYZ5rxDGUOnw== =8KKN -----END PGP SIGNATURE----- --Sig_/8EbDUZu_Pc=BJmIvJg7IYfq-- From debbugs-submit-bounces@debbugs.gnu.org Thu Nov 08 02:45:38 2018 Received: (at 33210) by debbugs.gnu.org; 8 Nov 2018 07:45:38 +0000 Received: from localhost ([127.0.0.1]:40224 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gKf0T-00079y-Rh for submit@debbugs.gnu.org; Thu, 08 Nov 2018 02:45:38 -0500 Received: from mail.lassieur.org ([83.152.10.219]:53922) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gKf0S-00079n-40 for 33210@debbugs.gnu.org; Thu, 08 Nov 2018 02:45:36 -0500 Received: from rodion (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id ca9988bd (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Thu, 8 Nov 2018 07:42:43 +0000 (UTC) References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <20181106011154.3f235763@scratchpost.org> <87efbzqbb0.fsf@lassieur.org> <20181106122036.25bad548@scratchpost.org> <875zxa5mfn.fsf@lassieur.org> <20181106211049.1469331b@scratchpost.org> <8736sd5d8z.fsf@lassieur.org> <20181108005941.519c464e@scratchpost.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Danny Milosavljevic Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode In-reply-to: <20181108005941.519c464e@scratchpost.org> Date: Thu, 08 Nov 2018 08:45:33 +0100 Message-ID: <878t24qag2.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210 Cc: 33210@debbugs.gnu.org, Andy Wingo X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Danny Milosavljevic writes: > Hi Cl=C3=A9ment, > >> > Yeah, right now users can query something using the web interface while >> > a build is updating (or running) at the cost of the returned data being >> > potentially very strange.=20=20 >>=20 >> This is quite unlikely. > > When testing the cuirass status frontend it happened regularily without > me trying to make it happen - took quite some time to find the cause, too. Maybe it happened without me noticing then. I'll trust you on this :-) >> No, there are no SQL requests during the build. There are some before >> the build, and some after the build. But they are all short. > > Yes, if there is one transaction right before starting the build and anot= her > transaction at the end of the build, then it's much better. > >> > I have to say I liked the external evaluator much more since then all >> > this complexity would be contained in the external program and it would >> > just magically work without special-casing any of this stuff.=20=20 >>=20 >> The evaluator is still external, I'm not sure what you are talking >> about. > > Hmm, I'll read through the source for a bit. I was of the impression > that now the cuirass main process did the updating of the build status > rather than the evaluator. Oh, yes, the updating of the build status is done by the main process, but the evaluator is still external and doesn't query the database. Why don't you like that? >> > It's difficult for guile-sqlite3 to handle SQLITE_BUSY correctly since >> > sqlite also uses SQLITE_BUSY to indicate errors that you are supposed = to >> > fail on.=20=20 >>=20 >> [...] >>=20 >> > Hmmmmmmmm. I think that can be done.=20=20 > > I've tried it and it works well enough, although some of the sqlite > documentation makes it sound like one cannot *just* retry some of the > calls (for example: sqlite3_step). > > It's one of the disadantages of fibers that every C library has to have > special code in it to support it somehow (if at all) - it means that > it has to be written in a way to make all calls non-blocking. Indeed. I wonder how goroutines deal with this. Cc'ing Andy. Cl=C3=A9ment From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 13 08:57:54 2018 Received: (at 33210-done) by debbugs.gnu.org; 13 Dec 2018 13:57:54 +0000 Received: from localhost ([127.0.0.1]:45951 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gXRUv-0007NY-My for submit@debbugs.gnu.org; Thu, 13 Dec 2018 08:57:53 -0500 Received: from mail.lassieur.org ([83.152.10.219]:48396) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gXRUt-0007NP-4W for 33210-done@debbugs.gnu.org; Thu, 13 Dec 2018 08:57:51 -0500 Received: from rodion (i577BC123.versanet.de [87.123.193.35]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id e6876c46 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Thu, 13 Dec 2018 13:57:48 +0000 (UTC) References: <87a7mvqikl.fsf@lassieur.org> <871s80o2zc.fsf@gnu.org> <87ftwgq7da.fsf@lassieur.org> <87ftwez2us.fsf@gnu.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#33210] Cuirass: Use a SQLite in single-thread mode In-reply-to: <87ftwez2us.fsf@gnu.org> Date: Thu, 13 Dec 2018 14:57:47 +0100 Message-ID: <87imzxttpg.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 33210-done Cc: 33210-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Hi Cl=C3=A9ment, > > Cl=C3=A9ment Lassieur skribis: > >> Ludovic Court=C3=A8s writes: >> >>> Hello, >>> >>> Cl=C3=A9ment Lassieur skribis: >>> >>>> These patches are supposed to slightly improve Cuirass' performances, >>>> because it doesn't use the multi-threading features. >>> >>> Did you notice a measurable difference? >> >> I haven't done any measurement yet, but according to the SQLite >> documentation: >> >> Setting -DSQLITE_THREADSAFE=3D0 causes all of the mutex and >> thread-safety logic in SQLite to be omitted. This is the single >> compile-time option that makes the most difference in optimizing the >> performance of SQLite. >> >> So even if the optimization is small, it's the option that has the >> biggest impact on performance. >> >>> We could do it, but IMO that should be a last resort because I=E2=80=99= d expect >>> it to be a micro-optimization. >> >> Lots of micro-optimizations lead to an overall faster application ;-). >> And this one doesn't make the code more complicated. To me it's just a >> bonus. > > I agree it doesn=E2=80=99t complicate the code; still, that=E2=80=99s a c= ouple of > additional package variants to deal with, for hardly measurable benefits > I suspect. > > I think we should focus on higher-level optimizations at this > development stage of Cuirass. For instance I have been meaning to patch > it so that it doesn=E2=80=99t have to process all the build logs, since it > doesn=E2=80=99t do anything with those logs and processing them involves = tons of > syscalls and string processing and introduces latency in fiber > scheduling. This is a simple change that could have a more visible > impact I believe. Hopefully I=E2=80=99ll get there real soon=E2=80=A6 Understood! Sorry to reply that late. Closing. Cl=C3=A9ment From unknown Thu Aug 14 22:17:28 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 11 Jan 2019 12:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator