From debbugs-submit-bounces@debbugs.gnu.org Sat Sep 29 16:36:09 2018 Received: (at submit) by debbugs.gnu.org; 29 Sep 2018 20:36:09 +0000 Received: from localhost ([127.0.0.1]:59216 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6LyD-0000bf-5g for submit@debbugs.gnu.org; Sat, 29 Sep 2018 16:36:09 -0400 Received: from eggs.gnu.org ([208.118.235.92]:52085) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6LyC-0000bR-5i for submit@debbugs.gnu.org; Sat, 29 Sep 2018 16:36:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g6Ly5-00069z-Mz for submit@debbugs.gnu.org; Sat, 29 Sep 2018 16:36:02 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:51848) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g6Ly5-00069r-IP for submit@debbugs.gnu.org; Sat, 29 Sep 2018 16:36:01 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33462) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g6Ly2-00049j-JC for guix-patches@gnu.org; Sat, 29 Sep 2018 16:36:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g6Lxx-00067d-JC for guix-patches@gnu.org; Sat, 29 Sep 2018 16:35:58 -0400 Received: from mail.lassieur.org ([83.152.10.219]:57670) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1g6Lxx-000674-6O for guix-patches@gnu.org; Sat, 29 Sep 2018 16:35:53 -0400 Received: from localhost.localdomain (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id 5f3510f9 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO) for ; Sat, 29 Sep 2018 20:33:05 +0000 (UTC) From: =?UTF-8?q?Cl=C3=A9ment=20Lassieur?= To: guix-patches@gnu.org Subject: [PATCH] database: Add builds only if one of their outputs is new. Date: Sat, 29 Sep 2018 22:35:32 +0200 Message-Id: <20180929203532.3826-1-clement@lassieur.org> X-Mailer: git-send-email 2.19.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.1 (-----) * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'. * src/cuirass/database.scm (db-add-output): New procedure. (db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and return #f if DB-ADD-OUTPUT returned an empty list. * src/schema.sql (Outputs): Set 'path' as primary key, instead of 'derivation, name'. * src/sql/upgrade-4.sql: New file with SQL queries to upgrade the database. * tests/database.scm (make-dummy-build): Use the #:OUTPUTS key. Get default OUTPUTS to depend on DRV. ("db-add-build-with-fixed-output"): New test. --- Makefile.am | 3 ++- src/cuirass/database.scm | 46 +++++++++++++++++++++++++++++----------- src/schema.sql | 3 +-- src/sql/upgrade-4.sql | 18 ++++++++++++++++ tests/database.scm | 16 ++++++++++++-- 5 files changed, 69 insertions(+), 17 deletions(-) create mode 100644 src/sql/upgrade-4.sql diff --git a/Makefile.am b/Makefile.am index 2f83659..7cea2ff 100644 --- a/Makefile.am +++ b/Makefile.am @@ -67,7 +67,8 @@ dist_pkgdata_DATA = src/schema.sql dist_sql_DATA = \ src/sql/upgrade-1.sql \ src/sql/upgrade-2.sql \ - src/sql/upgrade-3.sql + src/sql/upgrade-3.sql \ + src/sql/upgrade-4.sql dist_css_DATA = \ src/static/css/bootstrap.css \ diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm index 6777d28..9664f1b 100644 --- a/src/cuirass/database.scm +++ b/src/cuirass/database.scm @@ -425,12 +425,33 @@ string." (failed-other 3) (canceled 4)) +(define (db-add-output derivation output) + "Insert OUTPUT associated with DERIVATION. If an output with the same path +already exists, return #f." + (with-db-critical-section db + (catch 'sqlite-error + (lambda () + (match output + ((name . path) + (sqlite-exec db "\ +INSERT INTO Outputs (derivation, name, path) VALUES (" + derivation ", " name ", " path ");"))) + (last-insert-rowid db)) + (lambda (key who code message . rest) + ;; If we get a unique-constraint-failed error, that means we have + ;; already inserted the same output. That happens with fixed-output + ;; derivations. + (if (= code SQLITE_CONSTRAINT_PRIMARYKEY) + #f + (apply throw key who code rest)))))) + (define (db-add-build build) - "Store BUILD in database the database. BUILD eventual outputs are stored in -the OUTPUTS table." + "Store BUILD in database the database only if one of its outputs is new. +Return #f otherwise. BUILD outputs are stored in the OUTPUTS table." (with-db-critical-section db (catch 'sqlite-error (lambda () + (sqlite-exec db "BEGIN TRANSACTION;") (sqlite-exec db " INSERT INTO Builds (derivation, evaluation, job_name, system, nix_name, log, status, timestamp, starttime, stoptime) @@ -446,21 +467,22 @@ VALUES (" (or (assq-ref build #:timestamp) 0) ", " (or (assq-ref build #:starttime) 0) ", " (or (assq-ref build #:stoptime) 0) ");") - (let ((derivation (assq-ref build #:derivation))) - (for-each (lambda (output) - (match output - ((name . path) - (sqlite-exec db "\ -INSERT INTO Outputs (derivation, name, path) VALUES (" - derivation ", " name ", " path ");")))) - (assq-ref build #:outputs)) - derivation)) + (let* ((derivation (assq-ref build #:derivation)) + (outputs (assq-ref build #:outputs)) + (new-outputs (filter-map (cut db-add-output derivation <>) + outputs))) + (if (null? new-outputs) + (begin (sqlite-exec db "ROLLBACK;") + #f) + (begin (sqlite-exec db "COMMIT;") + derivation)))) (lambda (key who code message . rest) ;; If we get a unique-constraint-failed error, that means we have ;; already inserted the same build. That happens when several jobs ;; produce the same derivation, and we can ignore it. (if (= code SQLITE_CONSTRAINT_PRIMARYKEY) - #f + (begin (sqlite-exec db "ROLLBACK;") + #f) (apply throw key who code rest)))))) (define* (db-update-build-status! drv status #:key log-file) diff --git a/src/schema.sql b/src/schema.sql index bfc9ca7..a9e4a6a 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -46,8 +46,7 @@ CREATE TABLE Evaluations ( CREATE TABLE Outputs ( derivation TEXT NOT NULL, name TEXT NOT NULL, - path TEXT NOT NULL, - PRIMARY KEY (derivation, name), + path TEXT NOT NULL PRIMARY KEY, FOREIGN KEY (derivation) REFERENCES Builds (derivation) ); diff --git a/src/sql/upgrade-4.sql b/src/sql/upgrade-4.sql new file mode 100644 index 0000000..e567f03 --- /dev/null +++ b/src/sql/upgrade-4.sql @@ -0,0 +1,18 @@ +BEGIN TRANSACTION; + +ALTER TABLE Outputs RENAME TO tmp_Outputs; + +CREATE TABLE Outputs ( + derivation TEXT NOT NULL, + name TEXT NOT NULL, + path TEXT NOT NULL PRIMARY KEY, + FOREIGN KEY (derivation) REFERENCES Builds (derivation) +); + +INSERT OR IGNORE INTO Outputs (derivation, name, path) +SELECT derivation, name, path +FROM tmp_Outputs; + +DROP TABLE tmp_Outputs; + +COMMIT; diff --git a/tests/database.scm b/tests/database.scm index 21a12f4..d9dfe13 100644 --- a/tests/database.scm +++ b/tests/database.scm @@ -57,14 +57,15 @@ (define* (make-dummy-build drv #:optional (eval-id 42) - #:key (outputs '(("foo" . "/foo")))) + #:key (outputs + `(("foo" . ,(format #f "~a.output" drv))))) `((#:derivation . ,drv) (#:eval-id . ,eval-id) (#:job-name . "job") (#:system . "x86_64-linux") (#:nix-name . "foo") (#:log . "log") - (#:outputs . (("foo" . "/foo"))))) + (#:outputs . ,outputs))) (define-syntax-rule (with-temporary-database body ...) (call-with-temporary-output-file @@ -114,6 +115,17 @@ INSERT INTO Evaluations (specification, in_progress) VALUES (3, false);") ;; there, see . (db-add-build build))) + (test-equal "db-add-build-with-fixed-output" + #f + (let ((build1 (make-dummy-build "/fixed1.drv" + #:outputs '(("out" . "/fixed-output")))) + (build2 (make-dummy-build "/fixed2.drv" + #:outputs '(("out" . "/fixed-output"))))) + (db-add-build build1) + + ;; Should return #f because the outputs are the same. + (db-add-build build2))) + (test-equal "db-update-build-status!" (list (build-status scheduled) (build-status started) -- 2.19.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 30 15:34:22 2018 Received: (at 32879) by debbugs.gnu.org; 30 Sep 2018 19:34:22 +0000 Received: from localhost ([127.0.0.1]:60268 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6hTx-0003Km-W1 for submit@debbugs.gnu.org; Sun, 30 Sep 2018 15:34:22 -0400 Received: from eggs.gnu.org ([208.118.235.92]:36959) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6hTv-0003KZ-Tf for 32879@debbugs.gnu.org; Sun, 30 Sep 2018 15:34:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g6hTq-0003Yw-2Q for 32879@debbugs.gnu.org; Sun, 30 Sep 2018 15:34:14 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:41474) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g6hTp-0003Yo-VC; Sun, 30 Sep 2018 15:34:14 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=52812 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1g6hTp-0007WT-NI; Sun, 30 Sep 2018 15:34:13 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: =?utf-8?Q?Cl=C3=A9ment?= Lassieur Subject: Re: [bug#32879] [PATCH] database: Add builds only if one of their outputs is new. References: <20180929203532.3826-1-clement@lassieur.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 9 =?utf-8?Q?Vend=C3=A9miaire?= an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 30 Sep 2018 21:34:12 +0200 In-Reply-To: <20180929203532.3826-1-clement@lassieur.org> (=?utf-8?Q?=22Cl?= =?utf-8?Q?=C3=A9ment?= Lassieur"'s message of "Sat, 29 Sep 2018 22:35:32 +0200") Message-ID: <87wor2u6dn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 32879 Cc: 32879@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) Hello Cl=C3=A9ment, Cl=C3=A9ment Lassieur skribis: > * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'. > * src/cuirass/database.scm (db-add-output): New procedure. > (db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and return #= f if > DB-ADD-OUTPUT returned an empty list. > * src/schema.sql (Outputs): Set 'path' as primary key, instead of 'deriva= tion, > name'. > * src/sql/upgrade-4.sql: New file with SQL queries to upgrade the databas= e. > * tests/database.scm (make-dummy-build): Use the #:OUTPUTS key. Get defa= ult > OUTPUTS to depend on DRV. > ("db-add-build-with-fixed-output"): New test. What=E2=80=99s the rationale? I suppose having a simpler primary key for =E2=80=98Outputs=E2=80=99 might help performance? Thank you, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 30 16:41:43 2018 Received: (at 32879) by debbugs.gnu.org; 30 Sep 2018 20:41:43 +0000 Received: from localhost ([127.0.0.1]:60304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6iX9-0004vy-GA for submit@debbugs.gnu.org; Sun, 30 Sep 2018 16:41:43 -0400 Received: from mail.lassieur.org ([83.152.10.219]:59080) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6iX7-0004vp-8n for 32879@debbugs.gnu.org; Sun, 30 Sep 2018 16:41:41 -0400 Received: from rodion (88.191.118.83 [88.191.118.83]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id fcb30af4 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Sun, 30 Sep 2018 20:38:42 +0000 (UTC) References: <20180929203532.3826-1-clement@lassieur.org> <87wor2u6dn.fsf@gnu.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#32879] [PATCH] database: Add builds only if one of their outputs is new. In-reply-to: <87wor2u6dn.fsf@gnu.org> Date: Sun, 30 Sep 2018 22:41:38 +0200 Message-ID: <87k1n2pvjx.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 32879 Cc: 32879@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Hello Cl=C3=A9ment, > > Cl=C3=A9ment Lassieur skribis: > >> * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'. >> * src/cuirass/database.scm (db-add-output): New procedure. >> (db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and return = #f if >> DB-ADD-OUTPUT returned an empty list. >> * src/schema.sql (Outputs): Set 'path' as primary key, instead of 'deriv= ation, >> name'. >> * src/sql/upgrade-4.sql: New file with SQL queries to upgrade the databa= se. >> * tests/database.scm (make-dummy-build): Use the #:OUTPUTS key. Get def= ault >> OUTPUTS to depend on DRV. >> ("db-add-build-with-fixed-output"): New test. > > What=E2=80=99s the rationale? I suppose having a simpler primary key for > =E2=80=98Outputs=E2=80=99 might help performance? There is a slight performance and db size gain but the primary reason is to have a better idea of Cuirass' load when looking at the pending builds. There will be less (no?) 'fake' builds. The idea is that all builds should be real builds. Does that make sense? > Thank you, > Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 01 05:10:04 2018 Received: (at 32879) by debbugs.gnu.org; 1 Oct 2018 09:10:04 +0000 Received: from localhost ([127.0.0.1]:60682 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6uDM-0006ln-1K for submit@debbugs.gnu.org; Mon, 01 Oct 2018 05:10:04 -0400 Received: from mail.lassieur.org ([83.152.10.219]:59104) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g6uDJ-0006lF-G0 for 32879@debbugs.gnu.org; Mon, 01 Oct 2018 05:10:02 -0400 Received: from newt (smtp.parrot.biz [62.23.167.188]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id cb66a0c0 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Mon, 1 Oct 2018 09:06:57 +0000 (UTC) References: <20180929203532.3826-1-clement@lassieur.org> <87wor2u6dn.fsf@gnu.org> <87k1n2pvjx.fsf@lassieur.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#32879] [PATCH] database: Add builds only if one of their outputs is new. In-reply-to: <87k1n2pvjx.fsf@lassieur.org> Date: Mon, 01 Oct 2018 11:09:59 +0200 Message-ID: <87sh1qxcbc.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 32879 Cc: 32879@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Cl=C3=A9ment Lassieur writes: > Ludovic Court=C3=A8s writes: > >> Hello Cl=C3=A9ment, >> >> Cl=C3=A9ment Lassieur skribis: >> >>> * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'. >>> * src/cuirass/database.scm (db-add-output): New procedure. >>> (db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and return= #f if >>> DB-ADD-OUTPUT returned an empty list. >>> * src/schema.sql (Outputs): Set 'path' as primary key, instead of 'deri= vation, >>> name'. >>> * src/sql/upgrade-4.sql: New file with SQL queries to upgrade the datab= ase. >>> * tests/database.scm (make-dummy-build): Use the #:OUTPUTS key. Get de= fault >>> OUTPUTS to depend on DRV. >>> ("db-add-build-with-fixed-output"): New test. >> >> What=E2=80=99s the rationale? I suppose having a simpler primary key for >> =E2=80=98Outputs=E2=80=99 might help performance? > > There is a slight performance and db size gain but the primary reason is > to have a better idea of Cuirass' load when looking at the pending > builds. There will be less (no?) 'fake' builds. The idea is that all > builds should be real builds. Also, it would allow to be notified when someone pushes on master a commit that triggers too many builds. From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 02 05:08:29 2018 Received: (at 32879) by debbugs.gnu.org; 2 Oct 2018 09:08:30 +0000 Received: from localhost ([127.0.0.1]:33743 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g7GfN-0001lP-LT for submit@debbugs.gnu.org; Tue, 02 Oct 2018 05:08:29 -0400 Received: from eggs.gnu.org ([208.118.235.92]:35722) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g7GfL-0001lC-61 for 32879@debbugs.gnu.org; Tue, 02 Oct 2018 05:08:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g7GfB-0001yb-Oz for 32879@debbugs.gnu.org; Tue, 02 Oct 2018 05:08:22 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:45612) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g7GfB-0001yX-Kr; Tue, 02 Oct 2018 05:08:17 -0400 Received: from [193.50.110.166] (port=57388 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1g7GfA-0002x9-W0; Tue, 02 Oct 2018 05:08:17 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: =?utf-8?Q?Cl=C3=A9ment?= Lassieur Subject: Re: [bug#32879] [PATCH] database: Add builds only if one of their outputs is new. References: <20180929203532.3826-1-clement@lassieur.org> <87wor2u6dn.fsf@gnu.org> <87k1n2pvjx.fsf@lassieur.org> <87sh1qxcbc.fsf@lassieur.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 11 =?utf-8?Q?Vend=C3=A9miaire?= an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 02 Oct 2018 11:08:14 +0200 In-Reply-To: <87sh1qxcbc.fsf@lassieur.org> (=?utf-8?Q?=22Cl=C3=A9ment?= Lassieur"'s message of "Mon, 01 Oct 2018 11:09:59 +0200") Message-ID: <878t3gu35t.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 32879 Cc: 32879@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) Hello! Cl=C3=A9ment Lassieur skribis: > Cl=C3=A9ment Lassieur writes: > >> Ludovic Court=C3=A8s writes: >> >>> Hello Cl=C3=A9ment, >>> >>> Cl=C3=A9ment Lassieur skribis: >>> >>>> * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-4.sql'. >>>> * src/cuirass/database.scm (db-add-output): New procedure. >>>> (db-add-build): Call DB-ADD-OUTPUT, rollback the transaction and retur= n #f if >>>> DB-ADD-OUTPUT returned an empty list. >>>> * src/schema.sql (Outputs): Set 'path' as primary key, instead of 'der= ivation, >>>> name'. >>>> * src/sql/upgrade-4.sql: New file with SQL queries to upgrade the data= base. >>>> * tests/database.scm (make-dummy-build): Use the #:OUTPUTS key. Get d= efault >>>> OUTPUTS to depend on DRV. >>>> ("db-add-build-with-fixed-output"): New test. >>> >>> What=E2=80=99s the rationale? I suppose having a simpler primary key f= or >>> =E2=80=98Outputs=E2=80=99 might help performance? >> >> There is a slight performance and db size gain but the primary reason is >> to have a better idea of Cuirass' load when looking at the pending >> builds. There will be less (no?) 'fake' builds. The idea is that all >> builds should be real builds. > > Also, it would allow to be notified when someone pushes on master a > commit that triggers too many builds. That makes a lot of sense, thank you! IMO you can go ahead and push it. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 02 07:26:22 2018 Received: (at 32879-done) by debbugs.gnu.org; 2 Oct 2018 11:26:22 +0000 Received: from localhost ([127.0.0.1]:33856 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g7Ioo-00011b-In for submit@debbugs.gnu.org; Tue, 02 Oct 2018 07:26:22 -0400 Received: from mail.lassieur.org ([83.152.10.219]:59132) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g7Iol-00011Q-R5 for 32879-done@debbugs.gnu.org; Tue, 02 Oct 2018 07:26:20 -0400 Received: from newt (smtp.parrot.biz [62.23.167.188]) by mail.lassieur.org (OpenSMTPD) with ESMTPSA id f6e8fdd1 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Tue, 2 Oct 2018 11:23:04 +0000 (UTC) References: <20180929203532.3826-1-clement@lassieur.org> <87wor2u6dn.fsf@gnu.org> <87k1n2pvjx.fsf@lassieur.org> <87sh1qxcbc.fsf@lassieur.org> <878t3gu35t.fsf@gnu.org> User-agent: mu4e 1.0; emacs 26.1 From: =?utf-8?Q?Cl=C3=A9ment?= Lassieur To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#32879] [PATCH] database: Add builds only if one of their outputs is new. In-reply-to: <878t3gu35t.fsf@gnu.org> Date: Tue, 02 Oct 2018 13:26:17 +0200 Message-ID: <87efd88u92.fsf@lassieur.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 32879-done Cc: 32879-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, Ludovic Court=C3=A8s writes: > That makes a lot of sense, thank you! > > IMO you can go ahead and push it. Great! Thank you for the review. Cl=C3=A9ment From unknown Mon Aug 18 14:23:57 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 31 Oct 2018 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator