From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 30 06:08:39 2017 Received: (at submit) by debbugs.gnu.org; 30 Jul 2017 10:08:39 +0000 Received: from localhost ([127.0.0.1]:33622 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dbl9F-0004eM-SL for submit@debbugs.gnu.org; Sun, 30 Jul 2017 06:08:39 -0400 Received: from eggs.gnu.org ([208.118.235.92]:53414) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dbl9D-0004e9-8c for submit@debbugs.gnu.org; Sun, 30 Jul 2017 06:08:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dbl94-0000By-M1 for submit@debbugs.gnu.org; Sun, 30 Jul 2017 06:08:26 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:44740) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dbl94-0000Bp-Hh for submit@debbugs.gnu.org; Sun, 30 Jul 2017 06:08:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44346) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dbl90-0008NO-Sf for guix-patches@gnu.org; Sun, 30 Jul 2017 06:08:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dbl8x-000068-Di for guix-patches@gnu.org; Sun, 30 Jul 2017 06:08:18 -0400 Received: from mail-wr0-x22f.google.com ([2a00:1450:400c:c0c::22f]:34986) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dbl8w-0008W4-Up for guix-patches@gnu.org; Sun, 30 Jul 2017 06:08:15 -0400 Received: by mail-wr0-x22f.google.com with SMTP id k71so115326712wrc.2 for ; Sun, 30 Jul 2017 03:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=62czKf9AvoguZ+lYAxDp9HE+mgUs9UpGgWly9YPOT8w=; b=cEd5ZFeN38edKK5YNvnHF1ljbvdELOFTXaHDyL4SY7JkMFbbUXe1kTvOIcUVwQXE/q amxo5eahMlLhF3iETqEAuU8NmzaU9iWv9mSGioNzSrGXBKyRrzRDaMBrSE2GTh9V99Ax Bs1gET3YiPjBClQMq2FvzoMSHoso+OrwK2yaJSHToa8/Sj90IetspS3FNO3OLzocP3jf 6DD8YJcKrkCcVZKkKA1qZA3VBg6zdsw/l36Q0xRBYaab+YQ7hvF+W2kKFlE4mZvolsfv WKci9L77ROTceyMH4RQ3MirLjwxquiIOB6BbaO8NFOrD0UE/0P5batJyTi/KTuRvuacx E9XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=62czKf9AvoguZ+lYAxDp9HE+mgUs9UpGgWly9YPOT8w=; b=j7VoymzB6BryX09hPMxi6v9CyWorCscxdfDbzzAqEQam9xnA1YbwnKukq3PtFVriBc 0TGthnH4aFLYctmsylwXKFYWBVJ7kZo2JYDVPxBPekuyzmMZUs/9E3m312cS134tyzhA pah3X9CRQgYbVVTO9G2pL7VBjiJU9EN17BF8t/+tCQMadUrLxrW+Glsz+RRG3rFqAtGP 5hFjLY/oiNkSf/BCjMKTbpL8P44Q1rtrlWtF3HyJpoPGgT37ylsqYcWa6obSCKNW/3uj I8rT1ae7MjaIAQYK7c7B/HzXi/l1LYRKWz4FZ1u1UbJ2zv5C+t/wC+7hJvKcV2f34tqm eBQg== X-Gm-Message-State: AIVw110VqT7WuVOQNUWJafz1P0C2v8KAKsWlH/fksIWTHygMG2vZIuAf 72VSgfF/R1Yzj1Kq X-Received: by 10.223.169.203 with SMTP id b69mr4897969wrd.80.1501409291113; Sun, 30 Jul 2017 03:08:11 -0700 (PDT) Received: from localhost.localdomain (ABayonne-551-1-53-108.w90-11.abo.wanadoo.fr. [90.11.72.108]) by smtp.gmail.com with ESMTPSA id 70sm25955019wrm.62.2017.07.30.03.08.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Jul 2017 03:08:10 -0700 (PDT) From: Mathieu Othacehe To: guix-patches@gnu.org Subject: [PATCH] cuirass: add Hydra compatible HTTP API. Date: Sun, 30 Jul 2017 12:07:59 +0200 Message-Id: <20170730100759.17734-1-m.othacehe@gmail.com> X-Mailer: git-send-email 2.13.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit Cc: Mathieu Othacehe X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) * bin/evaluate.in (fill-job): New procedure. (main): Use it to fill informations (nix-name, system) that will later be added to database. * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API. (Database)[Derivation]: Add system and nix_name fields. (Database)[Builds]: Add id, status, timestamp, starttime and stoptime fields. Remove output field. (Database)[Outputs]: New table describing the build outputs. * src/cuirass/base.scm (build-packages): Add new fields to build object before adding it to database. * src/cuirass/database.scm (db-get-build, db-get-builds): New procedures to get a build by id from database and a list of builds using filter parameters respectively. * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it object->json-string. (object->json-scm): Move it utils.scm. (handle-*-request): New helpers procedures. (request-parameters): New procedure to parse a request query. (url-handler): Add new API's. * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported procedures moved from http.scm. * src/schema.sql (Outputs) : New table. (Derivations): Add system and nix_name columns. (Builds): Remove output column and add id, status, timestamp, starttime and stoptime columns. --- Hi, Here's a first draft adding partial support for Hydra API in Cuirass. It can be tested using curl or, better, with Emacs Guix. The following elisp will change hydra url to a local running Cuirass server. (setq guix-hydra-url "http://127.0.0.1:8080/") Then, it should be possible to use M-x guix-hydra-latest-builds. The commands guix-hydra-jobsets and guix-hydra-queued-builds won't function because the associated API's are not implemented yet. There's a problem with /build/:build-id/log/raw API because it is trying to fork while multiple threads are running (because of decompressed-port function). It seems to work but a warning message is printed. Thanks, Mathieu bin/evaluate.in | 18 +++- doc/cuirass.texi | 242 ++++++++++++++++++++++++++++++++++++++++++++++- src/cuirass/base.scm | 43 ++++++--- src/cuirass/database.scm | 139 ++++++++++++++++++++++++--- src/cuirass/http.scm | 137 +++++++++++++++++++++------ src/cuirass/utils.scm | 22 ++++- src/schema.sql | 17 +++- 7 files changed, 558 insertions(+), 60 deletions(-) diff --git a/bin/evaluate.in b/bin/evaluate.in index d1d0767..858c34e 100644 --- a/bin/evaluate.in +++ b/bin/evaluate.in @@ -28,9 +28,22 @@ exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@" (use-modules (cuirass) (ice-9 match) (ice-9 pretty-print) + (srfi srfi-26) (guix build utils) + (guix derivations) (guix store)) +(define (fill-job job eval-id) + "Given JOB assoc list, add EVAL-ID to it. Also process #:nix-name and + #:system from derivation stored in JOB." + (let ((drv (read-derivation-from-file + (assq-ref job #:derivation)))) + ((compose + (cut acons #:eval-id eval-id <>) + (cut acons #:nix-name (derivation-name drv) <>) + (cut acons #:system (derivation-system drv) <>)) + job))) + (define* (main #:optional (args (command-line))) (match args ((command load-path guix-package-path cachedir specstr database) @@ -73,8 +86,9 @@ exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@" (pretty-print (map (lambda (thunk) (let* ((job (call-with-time-display thunk)) - ;; Keep track of SPEC id in the returned jobs. - (job* (acons #:eval-id eval-id job))) + ;; Fill job with informations that will later be + ;; added to database. + (job* (fill-job job eval-id))) (db-add-derivation db job*) job*)) thunks) diff --git a/doc/cuirass.texi b/doc/cuirass.texi index 12bc02f..2392e2f 100644 --- a/doc/cuirass.texi +++ b/doc/cuirass.texi @@ -11,6 +11,7 @@ This manual is for Cuirass version @value{VERSION}, a build automation server. Copyright @copyright{} 2016, 2017 Mathieu Lirzin +Copyright @copyright{} 2017 Mathieu Othacehe @quotation Permission is granted to copy, distribute and/or modify this document @@ -56,6 +57,7 @@ Tutorial sections: Reference sections: * Invocation:: How to run Cuirass. * Database:: About the database schema. +* Web API:: Description of the Web API. * Contributing:: Your help needed! * GNU Free Documentation License:: The license of this manual. @@ -312,6 +314,13 @@ This field holds the @code{id} of an evaluation from the @item job_name This text field holds the name of the job. + +@item system +This text field holds the system name of the derivation. + +@item nix_name +This text field holds the name of the derivation. + @end table @section Builds @@ -322,6 +331,9 @@ that builds are not in a one to one relationship with derivations in order to keep track of non-deterministic compilations. @table @code +@item id +This is an automatically incrementing numeric identifier. + @item derivation This text field holds the absolute name of the derivation file that resulted in this build. @@ -334,11 +346,233 @@ belongs. @item log This text field holds the absolute file name of the build log file. -@item output -This text field holds the absolute directory name of the build output or -@code{NULL} if the build failed. +@item status +This integer field holds the build status of the derivation. + +@item timestamp +This integer field holds a timestamp taken at build creation time. + +@item starttime +This integer field holds a timestamp taken at build start time. +Currently, it has the same value as the @code{timestamp} above. + +@item stoptime +This integer field holds a timestamp taken at build stop time. +Currently, it has the same value as the @code{timestamp} above. + +@end table + +@section Outputs +@cindex outputs, database + +This table keep tracks for every eventual build outputs. Each build +stored in @code{Builds} table may have zero (if it has failed), one or +multiple outputs. + +@table @code +@item build +This field holds the @code{id} of a build from the +@code{Builds} table. + +@item name +This text field holds the name of the output. + +@item path +This text field holds the path of the output. + +@end table + +@c ********************************************************************* +@node Web API +@chapter Web API +@cindex web api + +Cuirass web API is derived from Hydra one, see @url{https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml, Hydra API description}. + +For now only a subset of this API is implemented. + +@section API description. +@cindex description, json + +@subsection Build informations. + +It is possible to query Cuirass web server for build informations. The +dedicated API is @code{"/build/:build-id"} where @code{build-id} is the +unique id associated to the build in database. + +For instance, querying a local Cuirass web server can be done with +@code{curl} and @code{jq} to format the JSON response : + +@example +$ curl -s "http://localhost:8080/build/2" | jq + +@{ + "id": 2, + "project": "guix", + "jobset": "master", + "job": "acpica-20150410-job", + "timestamp": 1501347493, + "starttime": 1501347493, + "stoptime": 1501347493, + "buildoutputs": @{ + "out": @{ + "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410" + @} + @}, + "system": "x86_64-linux", + "nixname": "acpica-20150410", + "buildstatus": 0, + "busy": 0, + "priority": 0, + "finished": 1, + "buildproducts": null, + "releasename": null, + "buildinputs_builds": null +@} +@end example + +If requested @code{build-id} is not known, the HTTP code 404 is +answered with a JSON error message. For example : + +@example +$ curl -s "http://localhost:8080/build/fff" + +@{"error" : "Build with ID fff doesn't exist."@} +@end example + +The nominal output is a JSON object whose fields are described +hereafter. + +@table @code +@item id +The unique build id. + +@item project +The associated specification name, as a string. + +@item jobset +The associated specification branch, as a string. + +@item job +The associated job-name, as a string. + +@item timestamp +Timestamp taken at build creation time. + +@item starttime +Timestamp taken at build start time. + +@item stoptime +Timestamp taken at build stop time. + +@item buildoutputs +Build outputs as a JSON object. The keys names are referring to the +eventual output names. The associated value is another JSON object which +only key is @code{path}. @code{path} value is the output directory in +store as a string. + +@item system +System name of the build, as a string. + +@item nixname +Derivation name, as a string. + +@item buildstatus +Build status, as an integer. Possible values are : + +@example +0 -> succeded +1 -> failed +2 -> failed dependency +3 -> failed other +4 -> cancelled +@end example + +@item busy +Whether the build is pending, as an integer (not implemented yet). + +@item priority +Build priority, as an integer (not implemented yet). + +@item finished +Build finished, as an integer (not implemented yet : always 1). + +@item buildproducts +Build products in store as a JSON object (not implemented yet). + +@item releasename +Unknown, not implemented yet. + +@item buildinputs_builds +Inputs used for the build, as a JSON object (not implemented yet). + @end table +@subsection Build raw log output. + +It is possible to ask Cuirass for the raw build output log with the API +@code{"/build/:build-id/log/raw"} where @code{build-id} is the +unique id associated to the build in database. + +The output is a raw text, for example : + +@example +$ curl http://localhost:8080/build/2/log/raw + +starting phase `set-SOURCE-DATE-EPOCH' +phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds +starting phase `set-paths' +... +@end example + +If requested @code{build-id} is not known, the HTTP code 404 is +answered with a JSON error message. For example : + +@example +$ curl -s "http://localhost:8080/build/fff/log/raw" + +@{"error" : "Build with ID fff doesn't exist."@} +@end example + +@subsection Latest builds. + +The list of latest builds can be obtained with the API +@code{"/api/latestbuilds"}. The output is a JSON array of +builds. Builds are represented as in @code{"/build/:build-id"} API. + +This request accepts a mandatory parameter and multiple optional ones. + +@table @code +@item nr +Limit query result to nr elements. This parameter is @emph{mandatory}. + +@item project +Filter query result to builds with the given @code{project}. + +@item jobset +Filter query result to builds with the given @code{jobset}. + +@item job +Filter query result to builds with the given @code{job} name. + +@item system +Filter query result to builds with the given @code{system}. + +@end table + +For example, to ask for the ten last builds : + +@example +$ curl "http://localhost:8080/api/latestbuilds?nr=10" +@end example + +or the five last builds which project is ``guix'' and jobset ``master' : + +@example +$ curl "http://localhost:8080/api/latestbuilds?nr=5&project=guix&jobset=master" +@end example + +If no builds matching given parameters are found and empty JSON array is returned. @c ********************************************************************* @node Contributing @@ -346,7 +580,7 @@ This text field holds the absolute directory name of the build output or Everyone is welcome to contribute to Cuirass. You can report bugs, send patches and share your ideas with others by sending emails the -@email{bug-cuirass@@framalistes.org, mailing list}. +@email{guix-devel@@gnu.org, mailing list}. Development is done using the Git distributed version control system. Thus, access to the repository is not strictly necessary. We welcome diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 326a530..15d2284 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -30,6 +30,7 @@ #:use-module (ice-9 popen) #:use-module (ice-9 rdelim) #:use-module (ice-9 receive) + #:use-module (srfi srfi-1) #:use-module (srfi srfi-19) #:use-module (srfi srfi-34) #:export (;; Procedures. @@ -154,25 +155,41 @@ directory and the sha1 of the top level commit in this directory." (define (build-packages store db jobs) "Build JOBS and return a list of Build results." + + (define hydra-build-status + ;; Build status as expected by hydra compatible API's. + '((succeeded . 0) + (failed . 1) + (failed-dependency . 2) + (failed-other . 3) + (cancelled . 4))) + (define (register job) (let* ((name (assq-ref job #:job-name)) (drv (assq-ref job #:derivation)) (eval-id (assq-ref job #:eval-id)) ;; XXX: How to keep logs from several attempts? (log (log-file store drv)) - (outputs (match (derivation-path->output-paths drv) - (((names . items) ...) - (filter (lambda (item) - (valid-path? store item)) - items))))) - (for-each (lambda (output) - (let ((build `((#:derivation . ,drv) - (#:eval-id . ,eval-id) - (#:log . ,log) - (#:output . ,output)))) - (db-add-build db build))) - outputs) - (format #t "~{~A ~}\n" outputs) + (outputs (filter-map (lambda (res) + (match res + ((name . path) + (and (valid-path? store path) + `(,name . ,path))))) + (derivation-path->output-paths drv))) + (cur-time (time-second (current-time time-utc)))) + (let ((build `((#:derivation . ,drv) + (#:eval-id . ,eval-id) + (#:log . ,log) + (#:status . + ,(match (length outputs) + (0 (assq-ref hydra-build-status 'failed)) + (_ (assq-ref hydra-build-status 'succeeded)))) + (#:outputs . ,outputs) + ;;; XXX: For now, we do not know start/stop build time. + (#:timestamp . ,cur-time) + (#:starttime . ,cur-time) + (#:stoptime . ,cur-time)))) + (db-add-build db build)) build)) ;; Pass all the jobs at once so we benefit from as much parallelism as diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm index 804b8c2..5f60fac 100644 --- a/src/cuirass/database.scm +++ b/src/cuirass/database.scm @@ -1,5 +1,6 @@ ;;; database.scm -- store evaluation and build results ;;; Copyright © 2016, 2017 Mathieu Lirzin +;;; Copyright © 2017 Mathieu Othacehe ;;; ;;; This file is part of Cuirass. ;;; @@ -21,6 +22,7 @@ #:use-module (cuirass utils) #:use-module (ice-9 match) #:use-module (ice-9 rdelim) + #:use-module (srfi srfi-1) #:use-module (sqlite3) #:export (;; Procedures. assq-refs @@ -35,6 +37,8 @@ db-add-derivation db-get-derivation db-add-build + db-get-build + db-get-builds read-sql-file read-quoted-string sqlite-exec @@ -147,10 +151,12 @@ INSERT OR IGNORE INTO Specifications (repo_name, url, load_path, file, \ (define (db-add-derivation db job) "Store a derivation result in database DB and return its ID." (sqlite-exec db "\ -INSERT OR IGNORE INTO Derivations (derivation, job_name, evaluation)\ - VALUES ('~A', '~A', '~A');" +INSERT OR IGNORE INTO Derivations (derivation, job_name, system, nix_name, evaluation)\ + VALUES ('~A', '~A', '~A', '~A', '~A');" (assq-ref job #:derivation) (assq-ref job #:job-name) + (assq-ref job #:system) + (assq-ref job #:nix-name) (assq-ref job #:eval-id))) (define (db-get-derivation db id) @@ -182,15 +188,126 @@ string." (else (loop (cons char chars))))))) (define (db-add-build db build) - "Store BUILD in database DB." - (sqlite-exec db "\ -INSERT INTO Builds (derivation, evaluation, log, output)\ - VALUES ('~A', '~A', '~A', '~A');" - (assq-ref build #:derivation) - (assq-ref build #:eval-id) - (assq-ref build #:log) - (assq-ref build #:output)) - (last-insert-rowid db)) + "Store BUILD in database DB. BUILS eventual outputs are stored +in the OUTPUTS table." + (let* ((build-exec + (sqlite-exec db "\ +INSERT INTO Builds (derivation, evaluation, log, status, timestamp, starttime, stoptime)\ + VALUES ('~A', '~A', '~A', '~A', '~A', '~A', '~A');" + (assq-ref build #:derivation) + (assq-ref build #:eval-id) + (assq-ref build #:log) + (assq-ref build #:status) + (assq-ref build #:timestamp) + (assq-ref build #:starttime) + (assq-ref build #:stoptime))) + (build-id (last-insert-rowid db))) + (for-each (lambda (output) + (match output + ((name . path) + (sqlite-exec db "\ +INSERT INTO Outputs (build, name, path) VALUES ('~A', '~A', '~A');" + build-id name path)))) + (assq-ref build #:outputs)) + build-id)) + +(define (db-get-outputs db build-id) + "Retrieve the OUTPUTS of the build identified by BUILD-ID in DB database." + (let loop ((rows + (sqlite-exec db "SELECT name, path FROM Outputs WHERE build='~A';" + build-id)) + (outputs '())) + (match rows + (() outputs) + ((#(name path) + . rest) + (loop rest + (cons `(,name . ((#:path . ,path))) + outputs)))))) + +(define db-build-request "\ +SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime, Builds.log, Builds.status,\ +Derivations.job_name, Derivations.system, Derivations.nix_name,\ +Specifications.repo_name, Specifications.branch \ +FROM Builds \ +INNER JOIN Derivations ON Builds.derivation = Derivations.derivation and Builds.evaluation = Derivations.evaluation \ +INNER JOIN Evaluations ON Derivations.evaluation = Evaluations.id \ +INNER JOIN Specifications ON Evaluations.specification = Specifications.repo_name") + +(define (db-format-build db build) + (match build + (#(id timestamp starttime stoptime log status job-name system + nix-name repo-name branch) + `((#:id . ,id) + (#:timestamp . ,timestamp) + (#:starttime . ,starttime) + (#:stoptime . ,stoptime) + (#:log . ,log) + (#:status . ,status) + (#:job-name . ,job-name) + (#:system . ,system) + (#:nix-name . ,nix-name) + (#:repo-name . ,repo-name) + (#:outputs . ,(db-get-outputs db id)) + (#:branch . ,branch))))) + +(define (db-get-build db id) + "Retrieve a build in database DB which corresponds to ID." + (let ((res (sqlite-exec db (string-append db-build-request + " WHERE Builds.id='~A';") id))) + (match res + ((build) + (db-format-build db build)) + (() #f)))) + +(define (db-get-builds db filters) + "Retrieve all builds in database DB which are matched by given FILTERS. +FILTERS is an assoc list which possible keys are 'project | 'jobset | 'job | +'system | 'nr." + + (define (format-where-clause filters) + (let ((where-clause + (filter-map + (lambda (param) + (match param + (('project project) + (format #f "Specifications.repo_name='~A'" project)) + (('jobset jobset) + (format #f "Specifications.branch='~A'" jobset)) + (('job job) + (format #f "Derivations.job_name='~A'" job)) + (('system system) + (format #f "Derivations.system='~A'" system)) + (_ #f))) + filters))) + (if (> (length where-clause) 0) + (string-append + "WHERE " + (string-join where-clause " AND ")) + ""))) + + (define (format-order-clause filters) + (any + (lambda (param) + (match param + (('nr number) + (format #f "ORDER BY Builds.id DESC LIMIT '~A';" number)) + (_ #f))) + filters)) + + (let loop ((rows + (sqlite-exec db (string-append + db-build-request + " " + (format-where-clause filters) + " " + (format-order-clause filters)))) + (outputs '())) + (match rows + (() outputs) + ((row . rest) + (loop rest + (cons (db-format-build db row) outputs)))))) (define (db-get-stamp db spec) "Return a stamp corresponding to specification SPEC in database DB." diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm index 33cd37b..976e24c 100644 --- a/src/cuirass/http.scm +++ b/src/cuirass/http.scm @@ -1,5 +1,6 @@ ;;;; http.scm -- HTTP API ;;; Copyright © 2016 Mathieu Lirzin +;;; Copyright © 2017 Mathieu Othacehe ;;; ;;; This file is part of Cuirass. ;;; @@ -19,52 +20,134 @@ (define-module (cuirass http) #:use-module (cuirass database) #:use-module (cuirass utils) - #:use-module (ice-9 hash-table) + #:use-module (guix build utils) + #:use-module (guix utils) #:use-module (ice-9 match) #:use-module (json) #:use-module (web request) #:use-module (web response) #:use-module (web server) #:use-module (web uri) - #:export (spec->json-string - run-cuirass-server)) + #:export (run-cuirass-server)) -;;; -;;; JSON format. -;;; +(define (build->hydra-build build) + "Convert BUILD to an assoc list matching hydra API format." + `((#:id . ,(assq-ref build #:id)) + (#:project . ,(assq-ref build #:repo-name)) + (#:jobset . ,(assq-ref build #:branch)) + (#:job . ,(assq-ref build #:job-name)) + (#:timestamp . ,(assq-ref build #:timestamp)) + (#:starttime . ,(assq-ref build #:starttime)) + (#:stoptime . ,(assq-ref build #:stoptime)) + (#:buildoutputs . ,(assq-ref build #:outputs)) + (#:system . ,(assq-ref build #:system)) + (#:nixname . ,(assq-ref build #:nix-name)) + (#:buildstatus . ,(assq-ref build #:status)) + + ;; TODO: Fill the fields above with correct values. + (#:busy . 0) + (#:priority . 0) + (#:finished . 1) + (#:buildproducts . #nil) + (#:releasename . #nil) + (#:buildinputs_builds . #nil))) + +(define (handle-build-request db build-id) + "Retrieve build identified by BUILD-ID in DB and convert it to hydra + format. Return #f is not build was found." + (let ((build (db-get-build db build-id))) + (and=> build build->hydra-build))) -(define (object->json-scm obj) - "Prepare OBJ for JSON usage." - (cond ((string? obj) obj) - ((number? obj) obj) - ((boolean? obj) obj) - ((null? obj) obj) - ((symbol? obj) (symbol->string obj)) - ((keyword? obj) (object->json-scm (keyword->symbol obj))) - ((alist? obj) (alist->hash-table (map object->json-scm obj))) - ((pair? obj) (cons (object->json-scm (car obj)) - (object->json-scm (cdr obj)))) - (else (object->string obj)))) - -(define* (spec->json-string spec #:key pretty) - "Return SPEC as a JSON object." - (scm->json-string (object->json-scm spec) #:pretty pretty)) +(define (handle-builds-request db filters) + "Retrieve all builds matched by FILTERS in DB and convert them to hydra + format." + (let ((builds (db-get-builds db filters))) + (map build->hydra-build builds))) + +(define (handle-log-request db build-id) + "Retrieve the log file of the build identified by BUILD-ID in DB. Return a + lambda which PORT argument is an input port from which the content of the + log file can be read or #f if the log file is not readable." + (let* ((build (db-get-build db build-id)) + (log (assq-ref build #:log)) + (access (and (string? log) + (access? log R_OK)))) + (and access + (lambda (port) + (dump-port (decompressed-port 'bzip2 (open-input-file log)) + port))))) + +(define (request-parameters request) + "Parse the REQUEST query parameters and return them under the form + '((parameter value) ...)." + (let* ((uri (request-uri request)) + (query (uri-query uri))) + (and query + (map (lambda (param) + (match (string-split param #\=) + ((key param) + (list (string->symbol key) param)))) + (string-split query #\&))))) ;;; ;;; Web server. ;;; +;;; The api is derived from the hydra one. It is partially described here : +;;; +;;; https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml +;;; (define (request-path-components request) (split-and-decode-uri-path (uri-path (request-uri request)))) (define (url-handler request body db) + (define* (respond response #:key body (db db)) (values response body db)) + + (define-syntax-rule (respond-json body ...) + (respond '((content-type . (application/json))) + #:body body ...)) + + (define-syntax-rule (respond-text body ...) + (respond '((content-type . (text/plain))) + #:body body ...)) + + (define-syntax-rule (respond-json-with-error error-code message) + (respond + (build-response #:headers '((content-type . (application/json))) + #:code error-code) + #:body + (object->json-string + `((error . ,message))))) + + (define (respond-build-not-found build) + (respond-json-with-error + 404 + (format #f "Build with ID ~a doesn't exist." build))) + (match (request-path-components request) (((or "jobsets" "specifications") . rest) - (respond '((content-type . (application/json))) - #:body (spec->json-string (car (db-get-specifications db))))) + (respond-json (object->json-string (car (db-get-specifications db))))) + (("build" build) + (let ((hydra-build (handle-build-request db build))) + (if hydra-build + (respond-json (object->json-string hydra-build)) + (respond-build-not-found build)))) + (("build" build "log" "raw") + (let ((log-response (handle-log-request db build))) + (if log-response + (respond-text log-response) + (respond-build-not-found build)))) + (("api" "latestbuilds") + (let* ((params (request-parameters request)) + ;; 'nr parameter is mandatory to limit query size. + (valid-params? (assq-ref params 'nr))) + (if valid-params? + (respond-json (object->json-string + (handle-builds-request db params))) + (respond-json-with-error 500 "Parameter not defined!")))) (_ (respond (build-response #:code 404) #:body (string-append "Resource not found: " @@ -73,6 +156,6 @@ (define* (run-cuirass-server db #:key (port 8080)) (format (current-error-port) "listening on port ~A~%" port) (run-server url-handler - 'http ;server implementation - `(#:port ,port) ;implementation parameters - db)) ;state + 'http + `(#:port ,port) + db)) diff --git a/src/cuirass/utils.scm b/src/cuirass/utils.scm index d966543..a932674 100644 --- a/src/cuirass/utils.scm +++ b/src/cuirass/utils.scm @@ -21,9 +21,29 @@ (define-module (cuirass utils) #:use-module (ice-9 match) #:use-module (srfi srfi-1) - #:export (alist?)) + #:use-module (json) + #:export (alist? + object->json-scm + object->json-string)) (define (alist? obj) "Return #t if OBJ is an alist." (and (list? obj) (every pair? obj))) + +(define (object->json-scm obj) + "Prepare OBJ for JSON usage." + (cond ((string? obj) obj) + ((number? obj) obj) + ((boolean? obj) obj) + ((null? obj) obj) + ((symbol? obj) (symbol->string obj)) + ((keyword? obj) (object->json-scm (keyword->symbol obj))) + ((alist? obj) (map object->json-scm obj)) + ((pair? obj) (cons (object->json-scm (car obj)) + (object->json-scm (cdr obj)))) + (else (object->string obj)))) + +(define* (object->json-string object #:key pretty) + "Return OBJECT as a JSON object." + (scm->json-string (object->json-scm object) #:pretty pretty)) diff --git a/src/schema.sql b/src/schema.sql index 329d89d..0ee428c 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -31,18 +31,31 @@ CREATE TABLE Derivations ( derivation TEXT NOT NULL, evaluation INTEGER NOT NULL, job_name TEXT NOT NULL, + system TEXT NOT NULL, + nix_name TEXT NOT NULL, PRIMARY KEY (derivation, evaluation), FOREIGN KEY (evaluation) REFERENCES Evaluations (id) ); +CREATE TABLE Outputs ( + build INTEGER NOT NULL, + name TEXT NOT NULL, + path TEXT NOT NULL, + PRIMARY KEY (build, name), + FOREIGN KEY (build) REFERENCES Builds (id) +); + -- Builds are not in a one to one relationship with derivations in order to -- keep track of non deterministic compilations. CREATE TABLE Builds ( + id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, derivation TEXT NOT NULL, evaluation INTEGER NOT NULL, log TEXT NOT NULL, - output TEXT, -- NULL if build failed - PRIMARY KEY (derivation, evaluation, output), + status INTEGER NOT NULL, + timestamp INTEGER NOT NULL, + starttime INTEGER NOT NULL, + stoptime INTEGER NOT NULL, FOREIGN KEY (derivation) REFERENCES Derivations (derivation), FOREIGN KEY (evaluation) REFERENCES Evaluations (id) ); -- 2.13.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 31 10:57:22 2017 Received: (at 27876) by debbugs.gnu.org; 31 Jul 2017 14:57:22 +0000 Received: from localhost ([127.0.0.1]:35485 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcC8H-00042U-Vz for submit@debbugs.gnu.org; Mon, 31 Jul 2017 10:57:22 -0400 Received: from eggs.gnu.org ([208.118.235.92]:49152) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcC8H-00042G-3G for 27876@debbugs.gnu.org; Mon, 31 Jul 2017 10:57:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dcC88-00075Z-LD for 27876@debbugs.gnu.org; Mon, 31 Jul 2017 10:57:15 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:54434) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcC88-00075L-Hg; Mon, 31 Jul 2017 10:57:12 -0400 Received: from [193.50.110.251] (port=37010 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dcC88-0001k9-2S; Mon, 31 Jul 2017 10:57:12 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH] cuirass: add Hydra compatible HTTP API. References: <20170730100759.17734-1-m.othacehe@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 13 Thermidor an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Mon, 31 Jul 2017 16:57:08 +0200 In-Reply-To: <20170730100759.17734-1-m.othacehe@gmail.com> (Mathieu Othacehe's message of "Sun, 30 Jul 2017 12:07:59 +0200") Message-ID: <87ini84ptn.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876 Cc: 27876@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hey! Mathieu Othacehe skribis: > * bin/evaluate.in (fill-job): New procedure. > (main): Use it to fill informations (nix-name, system) that will later be > added to database. > * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP A= PI. > (Database)[Derivation]: Add system and nix_name fields. > (Database)[Builds]: Add id, status, timestamp, starttime and stoptime > fields. Remove output field. > (Database)[Outputs]: New table describing the build outputs. > * src/cuirass/base.scm (build-packages): Add new fields to build object b= efore > adding it to database. > * src/cuirass/database.scm (db-get-build, db-get-builds): New procedures = to get > a build by id from database and a list of builds using filter parameters > respectively. > * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rena= me it > object->json-string. > (object->json-scm): Move it utils.scm. > (handle-*-request): New helpers procedures. > (request-parameters): New procedure to parse a request query. > (url-handler): Add new API's. > * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported > procedures moved from http.scm. > * src/schema.sql (Outputs) : New table. > (Derivations): Add system and nix_name columns. > (Builds): Remove output column and add id, status, timestamp, starttime a= nd > stoptime columns. > --- > > Hi, > > Here's a first draft adding partial support for Hydra API in Cuirass. > It can be tested using curl or, better, with Emacs Guix. > > The following elisp will change hydra url to a local running Cuirass serv= er. > > (setq guix-hydra-url "http://127.0.0.1:8080/") > > Then, it should be possible to use M-x guix-hydra-latest-builds. Woow! > The commands guix-hydra-jobsets and guix-hydra-queued-builds won't functi= on > because the associated API's are not implemented yet. > > There's a problem with /build/:build-id/log/raw API because it is trying = to fork > while multiple threads are running (because of decompressed-port function= ). It seems > to work but a warning message is printed. I think libbz2 and libz have the same API, so we could probably adapt (guix zlib) so that it supports libbz2 as well, which would solve this problem. That said, there=E2=80=99s only one place where we work, which is where we = spawn the =E2=80=98evaluate=E2=80=99 command. It may be that spawning it with = =E2=80=98open-pipe=E2=80=99 from (ice-9 popen) would sidestep the problem because =E2=80=98open-pipe=E2= =80=99 is specifically written to permit this. Some comments follow. > +@section API description. > +@cindex description, json > + > +@subsection Build informations. =E2=80=9CInformation=E2=80=9D is always singular (uncountable). Also, no t= railing period in section names. > +@example > +0 -> succeded ^ =E2=80=9Csucceeded=E2=80=9D > +@subsection Build raw log output. > + > +It is possible to ask Cuirass for the raw build output log with the API > +@code{"/build/:build-id/log/raw"} where @code{build-id} is the ^ ^^^^^^ Nitpick: we should probably write @var{build-id} (and remove the colon). > +unique id associated to the build in database. > + > +The output is a raw text, for example : > + > +@example > +$ curl http://localhost:8080/build/2/log/raw Initially I was thinking about having something similar in =E2=80=98guix publish=E2=80=99, but that would not include the =E2=80=9C/build=E2=80=9D p= art anyway, so it=E2=80=99s good to have this in Cuirass. > +(define (db-get-builds db filters) > + "Retrieve all builds in database DB which are matched by given FILTERS. > +FILTERS is an assoc list which possible keys are 'project | 'jobset | 'j= ob | > +'system | 'nr." Perhaps the database part of this change could have been a separate commit (first commit: store the relevant info in the DB and provide procedures to access it; second commit: implement the HTTP API to access the DB.) > + (("build" build) > + (let ((hydra-build (handle-build-request db build))) > + (if hydra-build > + (respond-json (object->json-string hydra-build)) > + (respond-build-not-found build)))) > + (("build" build "log" "raw") > + (let ((log-response (handle-log-request db build))) > + (if log-response > + (respond-text log-response) > + (respond-build-not-found build)))) > + (("api" "latestbuilds") > + (let* ((params (request-parameters request)) > + ;; 'nr parameter is mandatory to limit query size. > + (valid-params? (assq-ref params 'nr))) > + (if valid-params? > + (respond-json (object->json-string > + (handle-builds-request db params))) > + (respond-json-with-error 500 "Parameter not defined!")))) Nice. :-) I think it would be nice to have a couple of tests for the HTTP API. There are helpers in tests/publish.scm and (guix tests http) that could probably be borrowed here. WDYT? Anyway, awesome work! It=E2=80=99s a really important gap that you=E2=80= =99re filling here. In the future it would be nice to have an API to add jobsets, trigger an evaluation, things like that. That=E2=80=99s probably more difficult though because we=E2=80=99ll need an authentication mechanism. Also we should consider Fiberizing the whole thing eventually, so that Cuirass can actually perform all its activities concurrently. Thanks! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 01 15:48:20 2017 Received: (at 27876) by debbugs.gnu.org; 1 Aug 2017 19:48:20 +0000 Received: from localhost ([127.0.0.1]:37296 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcd9P-0001CJ-TR for submit@debbugs.gnu.org; Tue, 01 Aug 2017 15:48:20 -0400 Received: from mail-wr0-f172.google.com ([209.85.128.172]:38729) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcd9O-0001Bw-ES for 27876@debbugs.gnu.org; Tue, 01 Aug 2017 15:48:18 -0400 Received: by mail-wr0-f172.google.com with SMTP id f21so10733380wrf.5 for <27876@debbugs.gnu.org>; Tue, 01 Aug 2017 12:48:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:in-reply-to:date :message-id:mime-version:content-transfer-encoding; bh=Cu3Pg0xSEM4a5HStJKSyyAYov5kQHcGzQkqhNMiWOcI=; b=R3tureldOxFNVIfkg2ZIkM50ODU8oTM+g+a4n2qk8DvYFXAMumTXq4F+fPSaUrkbow ddMgUmj+QSXKIXtfY+UwVK2aQ5mIZizES++8Hw4dVj1yQiDLAjY70hZXeunHc7+ICnxz 0RHfvOMPndSotNtRWunoUNVxcATKv/3Y1ztWXMVbu1RsJUr+w3jHtni7O3GN9OfpTZz8 8ruTg8tdh9wWgD+6tFPKn262aFoPsc2HI/tLPf2fulPJXyzFuocPUH95QeI17EWQTY+s xJtGpXL5DJIGI8uybKuBa/BQoRNvhV7CxypZJY7Ip+JzKErjR72kqMV6CUKjkWiPH9Bh ShGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version:content-transfer-encoding; bh=Cu3Pg0xSEM4a5HStJKSyyAYov5kQHcGzQkqhNMiWOcI=; b=QjvU4sKsx+UWGjLSdhX0uDTyLvGt47/UbQ07mAIELxijtjDcD0HvNBZOR/9M8g1oO0 Y2oyGlROlcFfiTimHbQFdAlZFtdQwULA5r+GGaKQEI8rQQ51D1nxlqNBuMyTkjV+HvhO AyWH4SeLIjxfGTP21WazG8ME2mEg/dSNgkOPPpDcTVJ2uy2f61zDZoLtHElNix61F+aa E1TDF2LaHTdyq2mQTuF7Llhps60dQHU0ZYCIWPKpJH5J2uXAKEZ3hteyk0Q62pdUetFh kzzmR52luwCO+a0+PT7/AOSyrWsJTJ7gfKKbUSPAxU1YB7jfXVtt0XAVj+g7RigoXbR5 +HWQ== X-Gm-Message-State: AIVw110aYg6HIslQmH8QlWZM5mXrKti+Q+d+97qiZ2M8ALEvmWGRM2hZ 2dG26olvt3qzIU6R X-Received: by 10.223.150.148 with SMTP id u20mr15911568wrb.195.1501616892113; Tue, 01 Aug 2017 12:48:12 -0700 (PDT) Received: from cervin (ABayonne-551-1-53-108.w90-11.abo.wanadoo.fr. [90.11.72.108]) by smtp.gmail.com with ESMTPSA id i21sm8179339wmf.3.2017.08.01.12.48.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Aug 2017 12:48:11 -0700 (PDT) References: <20170730100759.17734-1-m.othacehe@gmail.com> <87ini84ptn.fsf@gnu.org> User-agent: mu4e 0.9.18; emacs 25.2.1 From: Mathieu Othacehe To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#27876] [PATCH] cuirass: add Hydra compatible HTTP API. In-reply-to: <87ini84ptn.fsf@gnu.org> Date: Tue, 01 Aug 2017 21:48:09 +0200 Message-ID: <877eynnk7a.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 27876 Cc: 27876@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi Ludo ! I'll submit a v2 soon, here are some answers to your remarks. > That said, there’s only one place where we work, which is where we spawn > the ‘evaluate’ command. It may be that spawning it with ‘open-pipe’ > from (ice-9 popen) would sidestep the problem because ‘open-pipe’ is > specifically written to permit this. The open-pipe solution seems fine indeed ! > I think it would be nice to have a couple of tests for the HTTP API. > There are helpers in tests/publish.scm and (guix tests http) that could > probably be borrowed here. WDYT? Sure, I'll add some new tests on the new API. > Anyway, awesome work! It’s a really important gap that you’re filling > here. Thank you ;) > In the future it would be nice to have an API to add jobsets, trigger an > evaluation, things like that. That’s probably more difficult though > because we’ll need an authentication mechanism. > > Also we should consider Fiberizing the whole thing eventually, so that > Cuirass can actually perform all its activities concurrently. Yes my planning is the following : 1. Fiberizing the whole thing to make it scalable. 2. Add build start/stop detection : I think the ideal would be wip-ui branch to get merged before, so that I can use the same mechanism to parse build output :) 3. Add authentification, and the other API you're describing. Still plenty of work !! Mathieu From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 01 15:51:40 2017 Received: (at 27876) by debbugs.gnu.org; 1 Aug 2017 19:51:40 +0000 Received: from localhost ([127.0.0.1]:37302 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcdCd-0001I4-Co for submit@debbugs.gnu.org; Tue, 01 Aug 2017 15:51:40 -0400 Received: from mail-wm0-f44.google.com ([74.125.82.44]:37433) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcdCa-0001Hd-NJ for 27876@debbugs.gnu.org; Tue, 01 Aug 2017 15:51:37 -0400 Received: by mail-wm0-f44.google.com with SMTP id t201so24041072wmt.0 for <27876@debbugs.gnu.org>; Tue, 01 Aug 2017 12:51:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7mQ2y6a3N0YxSWo9rbwmo0LU/mpoG/y9JOCXW+bbYLo=; b=RYWzbsWN2f7JizVnmn+V9+4/7kjvqdFRVPNFtJdvYrAICJWYyqgmiqEmJpcrLdpurq 0m7tytkglhLbP7P+FwXUqQf9zLFmfIXQnaVRtC4oCdql0GOlC8+TMzPSwXf2wViWldhX HRfM/AXyARLYfxKe4fyABsPRlp7eI7HxqgDdiR3+oagQwDTYCPFQK8QV0AYEOkBlZk0U t0z0QhTMP0/biG9Z0jkFEl8+Xc3PeWCrvE73DitsA2Ao1ZrYqUIXy8OiwXNIWEZLHC/L q+e8oPOAvYUsTk7R18xElONZ7npxoLt9QqVsdMN4gzCFx1dOzEEIQfsI5X02YlZUO90B a0eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7mQ2y6a3N0YxSWo9rbwmo0LU/mpoG/y9JOCXW+bbYLo=; b=N9fkCaMz01g1kyiywUMH8b9DE+02xlMwv10aBhRUzzvWietHxo50APbL1uZbTrg4/E MWscFKUTlojZM8yn6zJqx0Luteb4wTJL90A6OMjwBh6of68F9qFRWHuaLAB9nzhK4ZVb +nwI89p7mPffMpQr3Rexbc5DmpI0iwCGdSZZf4HthqN2/rjdN9OBDTBzKmx6Gs5t9G8P Hkgpx9WjEHlpMfsJEhpV+2ob60IIY5N4BlbWL+zpTz/9NEuGcWhaHs3NvQi6ZigU0mdI xxx05QzXEE/OsN6BC6v4/9HAdAaExL53/oGVd99pp116M7kSs5U1xjKOgZ2wKw1Egsfv Gmjw== X-Gm-Message-State: AIVw112g78peDVWnnC6LOLAW5MpYxWGyb58Qj00VfIeyPaO8QSaRk9CC erANVnifDvWSLqOp X-Received: by 10.28.174.7 with SMTP id x7mr2211798wme.43.1501617090533; Tue, 01 Aug 2017 12:51:30 -0700 (PDT) Received: from localhost.localdomain (ABayonne-551-1-53-108.w90-11.abo.wanadoo.fr. [90.11.72.108]) by smtp.gmail.com with ESMTPSA id e76sm2107696wmd.36.2017.08.01.12.51.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Aug 2017 12:51:29 -0700 (PDT) From: Mathieu Othacehe To: 27876@debbugs.gnu.org Subject: [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. Date: Tue, 1 Aug 2017 21:51:22 +0200 Message-Id: <20170801195124.7030-1-m.othacehe@gmail.com> X-Mailer: git-send-email 2.13.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 27876 Cc: Mathieu Othacehe X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) * bin/evaluate.in (fill-job): New procedure. (main): Use it to fill informations (nix-name, system) that will later be added to database. * doc/cuirass.texi (Database)[Derivation]: Add system and nix_name fields. (Database)[Builds]: Add id, status, timestamp, starttime and stoptime fields. Remove output field. (Database)[Outputs]: New table describing the build outputs. * src/cuirass/base.scm (build-packages): Add new fields to build object before adding it to database. * src/cuirass/database.scm (db-get-build, db-get-builds): New procedures to get a build by id from database and a list of builds using filter parameters respectively. * src/schema.sql (Outputs) : New table. (Derivations): Add system and nix_name columns. (Builds): Remove output column and add id, status, timestamp, starttime and stoptime columns. --- bin/evaluate.in | 18 +++++- doc/cuirass.texi | 49 ++++++++++++++++- src/cuirass/base.scm | 43 ++++++++++----- src/cuirass/database.scm | 139 +++++++++++++++++++++++++++++++++++++++++++---- src/schema.sql | 17 +++++- 5 files changed, 235 insertions(+), 31 deletions(-) diff --git a/bin/evaluate.in b/bin/evaluate.in index d1d0767..858c34e 100644 --- a/bin/evaluate.in +++ b/bin/evaluate.in @@ -28,9 +28,22 @@ exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@" (use-modules (cuirass) (ice-9 match) (ice-9 pretty-print) + (srfi srfi-26) (guix build utils) + (guix derivations) (guix store)) +(define (fill-job job eval-id) + "Given JOB assoc list, add EVAL-ID to it. Also process #:nix-name and + #:system from derivation stored in JOB." + (let ((drv (read-derivation-from-file + (assq-ref job #:derivation)))) + ((compose + (cut acons #:eval-id eval-id <>) + (cut acons #:nix-name (derivation-name drv) <>) + (cut acons #:system (derivation-system drv) <>)) + job))) + (define* (main #:optional (args (command-line))) (match args ((command load-path guix-package-path cachedir specstr database) @@ -73,8 +86,9 @@ exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@" (pretty-print (map (lambda (thunk) (let* ((job (call-with-time-display thunk)) - ;; Keep track of SPEC id in the returned jobs. - (job* (acons #:eval-id eval-id job))) + ;; Fill job with informations that will later be + ;; added to database. + (job* (fill-job job eval-id))) (db-add-derivation db job*) job*)) thunks) diff --git a/doc/cuirass.texi b/doc/cuirass.texi index 2899ffb..443b53c 100644 --- a/doc/cuirass.texi +++ b/doc/cuirass.texi @@ -11,6 +11,7 @@ This manual is for Cuirass version @value{VERSION}, a build automation server. Copyright @copyright{} 2016, 2017 Mathieu Lirzin +Copyright @copyright{} 2017 Mathieu Othacehe @quotation Permission is granted to copy, distribute and/or modify this document @@ -312,6 +313,13 @@ This field holds the @code{id} of an evaluation from the @item job_name This text field holds the name of the job. + +@item system +This text field holds the system name of the derivation. + +@item nix_name +This text field holds the name of the derivation. + @end table @section Builds @@ -322,6 +330,9 @@ that builds are not in a one to one relationship with derivations in order to keep track of non-deterministic compilations. @table @code +@item id +This is an automatically incrementing numeric identifier. + @item derivation This text field holds the absolute name of the derivation file that resulted in this build. @@ -334,9 +345,41 @@ belongs. @item log This text field holds the absolute file name of the build log file. -@item output -This text field holds the absolute directory name of the build output or -@code{NULL} if the build failed. +@item status +This integer field holds the build status of the derivation. + +@item timestamp +This integer field holds a timestamp taken at build creation time. + +@item starttime +This integer field holds a timestamp taken at build start time. +Currently, it has the same value as the @code{timestamp} above. + +@item stoptime +This integer field holds a timestamp taken at build stop time. +Currently, it has the same value as the @code{timestamp} above. + +@end table + +@section Outputs +@cindex outputs, database + +This table keep tracks for every eventual build outputs. Each build +stored in @code{Builds} table may have zero (if it has failed), one or +multiple outputs. + +@table @code +@item build +This field holds the @code{id} of a build from the +@code{Builds} table. + +@item name +This text field holds the name of the output. + +@item path +This text field holds the path of the output. + +@end table @end table diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 6abf871..8068539 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -31,6 +31,7 @@ #:use-module (ice-9 popen) #:use-module (ice-9 rdelim) #:use-module (ice-9 receive) + #:use-module (srfi srfi-1) #:use-module (srfi srfi-19) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -180,25 +181,41 @@ directory and the sha1 of the top level commit in this directory." (define (build-packages store db jobs) "Build JOBS and return a list of Build results." + + (define hydra-build-status + ;; Build status as expected by hydra compatible API's. + '((succeeded . 0) + (failed . 1) + (failed-dependency . 2) + (failed-other . 3) + (cancelled . 4))) + (define (register job) (let* ((name (assq-ref job #:job-name)) (drv (assq-ref job #:derivation)) (eval-id (assq-ref job #:eval-id)) ;; XXX: How to keep logs from several attempts? (log (log-file store drv)) - (outputs (match (derivation-path->output-paths drv) - (((names . items) ...) - (filter (lambda (item) - (valid-path? store item)) - items))))) - (for-each (lambda (output) - (let ((build `((#:derivation . ,drv) - (#:eval-id . ,eval-id) - (#:log . ,log) - (#:output . ,output)))) - (db-add-build db build))) - outputs) - (format #t "~{~A ~}\n" outputs) + (outputs (filter-map (lambda (res) + (match res + ((name . path) + (and (valid-path? store path) + `(,name . ,path))))) + (derivation-path->output-paths drv))) + (cur-time (time-second (current-time time-utc)))) + (let ((build `((#:derivation . ,drv) + (#:eval-id . ,eval-id) + (#:log . ,log) + (#:status . + ,(match (length outputs) + (0 (assq-ref hydra-build-status 'failed)) + (_ (assq-ref hydra-build-status 'succeeded)))) + (#:outputs . ,outputs) + ;;; XXX: For now, we do not know start/stop build time. + (#:timestamp . ,cur-time) + (#:starttime . ,cur-time) + (#:stoptime . ,cur-time)))) + (db-add-build db build)) build)) ;; Pass all the jobs at once so we benefit from as much parallelism as diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm index 804b8c2..5f60fac 100644 --- a/src/cuirass/database.scm +++ b/src/cuirass/database.scm @@ -1,5 +1,6 @@ ;;; database.scm -- store evaluation and build results ;;; Copyright © 2016, 2017 Mathieu Lirzin +;;; Copyright © 2017 Mathieu Othacehe ;;; ;;; This file is part of Cuirass. ;;; @@ -21,6 +22,7 @@ #:use-module (cuirass utils) #:use-module (ice-9 match) #:use-module (ice-9 rdelim) + #:use-module (srfi srfi-1) #:use-module (sqlite3) #:export (;; Procedures. assq-refs @@ -35,6 +37,8 @@ db-add-derivation db-get-derivation db-add-build + db-get-build + db-get-builds read-sql-file read-quoted-string sqlite-exec @@ -147,10 +151,12 @@ INSERT OR IGNORE INTO Specifications (repo_name, url, load_path, file, \ (define (db-add-derivation db job) "Store a derivation result in database DB and return its ID." (sqlite-exec db "\ -INSERT OR IGNORE INTO Derivations (derivation, job_name, evaluation)\ - VALUES ('~A', '~A', '~A');" +INSERT OR IGNORE INTO Derivations (derivation, job_name, system, nix_name, evaluation)\ + VALUES ('~A', '~A', '~A', '~A', '~A');" (assq-ref job #:derivation) (assq-ref job #:job-name) + (assq-ref job #:system) + (assq-ref job #:nix-name) (assq-ref job #:eval-id))) (define (db-get-derivation db id) @@ -182,15 +188,126 @@ string." (else (loop (cons char chars))))))) (define (db-add-build db build) - "Store BUILD in database DB." - (sqlite-exec db "\ -INSERT INTO Builds (derivation, evaluation, log, output)\ - VALUES ('~A', '~A', '~A', '~A');" - (assq-ref build #:derivation) - (assq-ref build #:eval-id) - (assq-ref build #:log) - (assq-ref build #:output)) - (last-insert-rowid db)) + "Store BUILD in database DB. BUILS eventual outputs are stored +in the OUTPUTS table." + (let* ((build-exec + (sqlite-exec db "\ +INSERT INTO Builds (derivation, evaluation, log, status, timestamp, starttime, stoptime)\ + VALUES ('~A', '~A', '~A', '~A', '~A', '~A', '~A');" + (assq-ref build #:derivation) + (assq-ref build #:eval-id) + (assq-ref build #:log) + (assq-ref build #:status) + (assq-ref build #:timestamp) + (assq-ref build #:starttime) + (assq-ref build #:stoptime))) + (build-id (last-insert-rowid db))) + (for-each (lambda (output) + (match output + ((name . path) + (sqlite-exec db "\ +INSERT INTO Outputs (build, name, path) VALUES ('~A', '~A', '~A');" + build-id name path)))) + (assq-ref build #:outputs)) + build-id)) + +(define (db-get-outputs db build-id) + "Retrieve the OUTPUTS of the build identified by BUILD-ID in DB database." + (let loop ((rows + (sqlite-exec db "SELECT name, path FROM Outputs WHERE build='~A';" + build-id)) + (outputs '())) + (match rows + (() outputs) + ((#(name path) + . rest) + (loop rest + (cons `(,name . ((#:path . ,path))) + outputs)))))) + +(define db-build-request "\ +SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime, Builds.log, Builds.status,\ +Derivations.job_name, Derivations.system, Derivations.nix_name,\ +Specifications.repo_name, Specifications.branch \ +FROM Builds \ +INNER JOIN Derivations ON Builds.derivation = Derivations.derivation and Builds.evaluation = Derivations.evaluation \ +INNER JOIN Evaluations ON Derivations.evaluation = Evaluations.id \ +INNER JOIN Specifications ON Evaluations.specification = Specifications.repo_name") + +(define (db-format-build db build) + (match build + (#(id timestamp starttime stoptime log status job-name system + nix-name repo-name branch) + `((#:id . ,id) + (#:timestamp . ,timestamp) + (#:starttime . ,starttime) + (#:stoptime . ,stoptime) + (#:log . ,log) + (#:status . ,status) + (#:job-name . ,job-name) + (#:system . ,system) + (#:nix-name . ,nix-name) + (#:repo-name . ,repo-name) + (#:outputs . ,(db-get-outputs db id)) + (#:branch . ,branch))))) + +(define (db-get-build db id) + "Retrieve a build in database DB which corresponds to ID." + (let ((res (sqlite-exec db (string-append db-build-request + " WHERE Builds.id='~A';") id))) + (match res + ((build) + (db-format-build db build)) + (() #f)))) + +(define (db-get-builds db filters) + "Retrieve all builds in database DB which are matched by given FILTERS. +FILTERS is an assoc list which possible keys are 'project | 'jobset | 'job | +'system | 'nr." + + (define (format-where-clause filters) + (let ((where-clause + (filter-map + (lambda (param) + (match param + (('project project) + (format #f "Specifications.repo_name='~A'" project)) + (('jobset jobset) + (format #f "Specifications.branch='~A'" jobset)) + (('job job) + (format #f "Derivations.job_name='~A'" job)) + (('system system) + (format #f "Derivations.system='~A'" system)) + (_ #f))) + filters))) + (if (> (length where-clause) 0) + (string-append + "WHERE " + (string-join where-clause " AND ")) + ""))) + + (define (format-order-clause filters) + (any + (lambda (param) + (match param + (('nr number) + (format #f "ORDER BY Builds.id DESC LIMIT '~A';" number)) + (_ #f))) + filters)) + + (let loop ((rows + (sqlite-exec db (string-append + db-build-request + " " + (format-where-clause filters) + " " + (format-order-clause filters)))) + (outputs '())) + (match rows + (() outputs) + ((row . rest) + (loop rest + (cons (db-format-build db row) outputs)))))) (define (db-get-stamp db spec) "Return a stamp corresponding to specification SPEC in database DB." diff --git a/src/schema.sql b/src/schema.sql index 329d89d..0ee428c 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -31,18 +31,31 @@ CREATE TABLE Derivations ( derivation TEXT NOT NULL, evaluation INTEGER NOT NULL, job_name TEXT NOT NULL, + system TEXT NOT NULL, + nix_name TEXT NOT NULL, PRIMARY KEY (derivation, evaluation), FOREIGN KEY (evaluation) REFERENCES Evaluations (id) ); +CREATE TABLE Outputs ( + build INTEGER NOT NULL, + name TEXT NOT NULL, + path TEXT NOT NULL, + PRIMARY KEY (build, name), + FOREIGN KEY (build) REFERENCES Builds (id) +); + -- Builds are not in a one to one relationship with derivations in order to -- keep track of non deterministic compilations. CREATE TABLE Builds ( + id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, derivation TEXT NOT NULL, evaluation INTEGER NOT NULL, log TEXT NOT NULL, - output TEXT, -- NULL if build failed - PRIMARY KEY (derivation, evaluation, output), + status INTEGER NOT NULL, + timestamp INTEGER NOT NULL, + starttime INTEGER NOT NULL, + stoptime INTEGER NOT NULL, FOREIGN KEY (derivation) REFERENCES Derivations (derivation), FOREIGN KEY (evaluation) REFERENCES Evaluations (id) ); -- 2.13.2 From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 01 15:51:40 2017 Received: (at 27876) by debbugs.gnu.org; 1 Aug 2017 19:51:41 +0000 Received: from localhost ([127.0.0.1]:37304 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcdCe-0001I6-7R for submit@debbugs.gnu.org; Tue, 01 Aug 2017 15:51:40 -0400 Received: from mail-wm0-f53.google.com ([74.125.82.53]:38074) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcdCb-0001Hg-W0 for 27876@debbugs.gnu.org; Tue, 01 Aug 2017 15:51:38 -0400 Received: by mail-wm0-f53.google.com with SMTP id m85so23985688wma.1 for <27876@debbugs.gnu.org>; Tue, 01 Aug 2017 12:51:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MMXaRww6+FI4CKPRShejRiJXfLxw7hSGbSnmEhduxb0=; b=Cr3tGQBouey0+p5tjGkDXEFMC8cEZVaG2+XKsRAzX/AClQYxKcm8gWovimQaEQxNwJ us0JpuSu5NYB3gHJTtqTCrzEz1NQVXoh2urLrJqRbQMDfdhaNlw6S4ZCpvJxQevLzews 9WCK0cSvwmy4nstlU6nTdR0/w4bRNS0zHpVryuLaJbLDXabcy/gDnyLcmMAH4OODUbnG CPhOK7JnrGIvDYbTn9iSGGBEEnJ9lPOrS/0Ws/BsISLrsJ3pENnYpv4M+70RxbNcpOxr CeCvQSH4w+vybmUvsPj5yoMY0ewHY/5ZC9wqfgkrzze2spI4otbxU164ZRAVMGj6bHDG tPkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MMXaRww6+FI4CKPRShejRiJXfLxw7hSGbSnmEhduxb0=; b=hIeBg8AKqFU2C3aoVUmXSSTiuWBLnwSRYnQAiMngsCcMZhk4JhXcjQjIXpAc6XABMw u0BqAlUOwuJddxe5o4+NhftaQs1wRtMGU0XiQLjgQSXCuLL2dWXSPBaK2uihUpuEmggO ZXb2yy8MMFBzt0T+9v61JOUlej3uKZB5Fd3lkXI0Sk3GXwa0o4aeLqf+HaVZgx/yIv6U rs/fxG4XolTJr9qsJmocEBdrnZs1YJe5NMQ+LIdYkZdUIkFcQQpwvN9zV93OtbXpkWpp j7WLABthtIMAYzDF+7brLfvsPneYhALEPv4b4JqWyM18ThO28q3cOZSSzgCOpOZJkTYn vqYQ== X-Gm-Message-State: AIVw112ZbZ5KmgrIfQEdZYrpZhzaaA3b5QY9rWuYt9JZlP8/6ngnUfl9 fy7iB2ebrG6JtU+X X-Received: by 10.28.109.27 with SMTP id i27mr2067870wmc.173.1501617091668; Tue, 01 Aug 2017 12:51:31 -0700 (PDT) Received: from localhost.localdomain (ABayonne-551-1-53-108.w90-11.abo.wanadoo.fr. [90.11.72.108]) by smtp.gmail.com with ESMTPSA id e76sm2107696wmd.36.2017.08.01.12.51.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Aug 2017 12:51:31 -0700 (PDT) From: Mathieu Othacehe To: 27876@debbugs.gnu.org Subject: [PATCH v2 2/3] cuirass: add Hydra compatible HTTP API. Date: Tue, 1 Aug 2017 21:51:23 +0200 Message-Id: <20170801195124.7030-2-m.othacehe@gmail.com> X-Mailer: git-send-email 2.13.2 In-Reply-To: <20170801195124.7030-1-m.othacehe@gmail.com> References: <20170801195124.7030-1-m.othacehe@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 27876 Cc: Mathieu Othacehe X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API. * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it object->json-string. (object->json-scm): Move it utils.scm. (handle-*-request): New helpers procedures. (request-parameters): New procedure to parse a request query. (url-handler): Add new API's. * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported procedures moved from http.scm. --- doc/cuirass.texi | 191 +++++++++++++++++++++++++++++++++++++++++++++++ src/cuirass/database.scm | 29 +++---- src/cuirass/http.scm | 150 ++++++++++++++++++++++++++++++------- src/cuirass/utils.scm | 22 +++++- 4 files changed, 350 insertions(+), 42 deletions(-) diff --git a/doc/cuirass.texi b/doc/cuirass.texi index 443b53c..bac2e54 100644 --- a/doc/cuirass.texi +++ b/doc/cuirass.texi @@ -57,6 +57,7 @@ Tutorial sections: Reference sections: * Invocation:: How to run Cuirass. * Database:: About the database schema. +* Web API:: Description of the Web API. * Contributing:: Your help needed! * GNU Free Documentation License:: The license of this manual. @@ -380,8 +381,198 @@ This text field holds the name of the output. This text field holds the path of the output. @end table + +@c ********************************************************************* +@node Web API +@chapter Web API +@cindex web api + +Cuirass web API is derived from Hydra one, see @url{https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml, Hydra API description}. + +For now only a subset of this API is implemented. + +@section API description +@cindex description, json + +@subsection Build information + +It is possible to query Cuirass web server for build informations. The +dedicated API is "/build/@var{build-id}" where @var{build-id} is the +unique id associated to the build in database. + +For instance, querying a local Cuirass web server can be done with +@code{curl} and @code{jq} to format the JSON response : + +@example +$ curl -s "http://localhost:8080/build/2" | jq + +@{ + "id": 2, + "project": "guix", + "jobset": "master", + "job": "acpica-20150410-job", + "timestamp": 1501347493, + "starttime": 1501347493, + "stoptime": 1501347493, + "buildoutputs": @{ + "out": @{ + "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410" + @} + @}, + "system": "x86_64-linux", + "nixname": "acpica-20150410", + "buildstatus": 0, + "busy": 0, + "priority": 0, + "finished": 1, + "buildproducts": null, + "releasename": null, + "buildinputs_builds": null +@} +@end example + +If requested @var{build-id} is not known, the HTTP code 404 is +answered with a JSON error message. For example : + +@example +$ curl -s "http://localhost:8080/build/fff" + +@{"error" : "Build with ID fff doesn't exist."@} +@end example + +The nominal output is a JSON object whose fields are described +hereafter. + +@table @code +@item id +The unique build id. + +@item project +The associated specification name, as a string. + +@item jobset +The associated specification branch, as a string. + +@item job +The associated job-name, as a string. + +@item timestamp +Timestamp taken at build creation time. + +@item starttime +Timestamp taken at build start time. + +@item stoptime +Timestamp taken at build stop time. + +@item buildoutputs +Build outputs as a JSON object. The keys names are referring to the +eventual output names. The associated value is another JSON object which +only key is @code{path}. @code{path} value is the output directory in +store as a string. + +@item system +System name of the build, as a string. + +@item nixname +Derivation name, as a string. + +@item buildstatus +Build status, as an integer. Possible values are : + +@example +0 -> succeeded +1 -> failed +2 -> failed dependency +3 -> failed other +4 -> cancelled +@end example + +@item busy +Whether the build is pending, as an integer (not implemented yet). + +@item priority +Build priority, as an integer (not implemented yet). + +@item finished +Build finished, as an integer (not implemented yet : always 1). + +@item buildproducts +Build products in store as a JSON object (not implemented yet). + +@item releasename +Unknown, not implemented yet. + +@item buildinputs_builds +Inputs used for the build, as a JSON object (not implemented yet). + +@end table + +@subsection Build raw log output + +It is possible to ask Cuirass for the raw build output log with the API +"/build/@var{build-id}/log/raw" where @var{build-id} is the +unique id associated to the build in database. + +The output is a raw text, for example : + +@example +$ curl http://localhost:8080/build/2/log/raw + +starting phase `set-SOURCE-DATE-EPOCH' +phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds +starting phase `set-paths' +... +@end example + +If requested @var{build-id} is not known, the HTTP code 404 is +answered with a JSON error message. For example : + +@example +$ curl -s "http://localhost:8080/build/fff/log/raw" + +@{"error" : "Build with ID fff doesn't exist."@} +@end example + +@subsection Latest builds + +The list of latest builds can be obtained with the API +"/api/latestbuilds". The output is a JSON array of +builds. Builds are represented as in "/build/@var{build-id} API. + +This request accepts a mandatory parameter and multiple optional ones. + +@table @code +@item nr +Limit query result to nr elements. This parameter is @emph{mandatory}. + +@item project +Filter query result to builds with the given @code{project}. + +@item jobset +Filter query result to builds with the given @code{jobset}. + +@item job +Filter query result to builds with the given @code{job} name. + +@item system +Filter query result to builds with the given @code{system}. + @end table +For example, to ask for the ten last builds : + +@example +$ curl "http://localhost:8080/api/latestbuilds?nr=10" +@end example + +or the five last builds which project is ``guix'' and jobset ``master' : + +@example +$ curl "http://localhost:8080/api/latestbuilds?nr=5&project=guix&jobset=master" +@end example + +If no builds matching given parameters are found and empty JSON array is returned. @c ********************************************************************* @node Contributing diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm index 5f60fac..6b84a48 100644 --- a/src/cuirass/database.scm +++ b/src/cuirass/database.scm @@ -226,7 +226,7 @@ INSERT INTO Outputs (build, name, path) VALUES ('~A', '~A', '~A');" outputs)))))) (define db-build-request "\ -SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime, Builds.log, Builds.status,\ +SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime, Builds.log, Builds.status, Builds.derivation,\ Derivations.job_name, Derivations.system, Derivations.nix_name,\ Specifications.repo_name, Specifications.branch \ FROM Builds \ @@ -236,20 +236,21 @@ INNER JOIN Specifications ON Evaluations.specification = Specifications.repo_nam (define (db-format-build db build) (match build - (#(id timestamp starttime stoptime log status job-name system + (#(id timestamp starttime stoptime log status derivation job-name system nix-name repo-name branch) - `((#:id . ,id) - (#:timestamp . ,timestamp) - (#:starttime . ,starttime) - (#:stoptime . ,stoptime) - (#:log . ,log) - (#:status . ,status) - (#:job-name . ,job-name) - (#:system . ,system) - (#:nix-name . ,nix-name) - (#:repo-name . ,repo-name) - (#:outputs . ,(db-get-outputs db id)) - (#:branch . ,branch))))) + `((#:id . ,id) + (#:timestamp . ,timestamp) + (#:starttime . ,starttime) + (#:stoptime . ,stoptime) + (#:log . ,log) + (#:status . ,status) + (#:derivation . ,derivation) + (#:job-name . ,job-name) + (#:system . ,system) + (#:nix-name . ,nix-name) + (#:repo-name . ,repo-name) + (#:outputs . ,(db-get-outputs db id)) + (#:branch . ,branch))))) (define (db-get-build db id) "Retrieve a build in database DB which corresponds to ID." diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm index 33cd37b..23c3ad7 100644 --- a/src/cuirass/http.scm +++ b/src/cuirass/http.scm @@ -1,5 +1,6 @@ ;;;; http.scm -- HTTP API ;;; Copyright © 2016 Mathieu Lirzin +;;; Copyright © 2017 Mathieu Othacehe ;;; ;;; This file is part of Cuirass. ;;; @@ -19,52 +20,147 @@ (define-module (cuirass http) #:use-module (cuirass database) #:use-module (cuirass utils) - #:use-module (ice-9 hash-table) + #:use-module (guix config) + #:use-module (guix build utils) + #:use-module (guix utils) #:use-module (ice-9 match) + #:use-module (ice-9 popen) #:use-module (json) #:use-module (web request) #:use-module (web response) #:use-module (web server) #:use-module (web uri) - #:export (spec->json-string - run-cuirass-server)) + #:export (run-cuirass-server)) -;;; -;;; JSON format. -;;; +(define (build->hydra-build build) + "Convert BUILD to an assoc list matching hydra API format." + `((#:id . ,(assq-ref build #:id)) + (#:project . ,(assq-ref build #:repo-name)) + (#:jobset . ,(assq-ref build #:branch)) + (#:job . ,(assq-ref build #:job-name)) + (#:timestamp . ,(assq-ref build #:timestamp)) + (#:starttime . ,(assq-ref build #:starttime)) + (#:stoptime . ,(assq-ref build #:stoptime)) + (#:buildoutputs . ,(assq-ref build #:outputs)) + (#:system . ,(assq-ref build #:system)) + (#:nixname . ,(assq-ref build #:nix-name)) + (#:buildstatus . ,(assq-ref build #:status)) + + ;; TODO: Fill the fields above with correct values. + (#:busy . 0) + (#:priority . 0) + (#:finished . 1) + (#:buildproducts . #nil) + (#:releasename . #nil) + (#:buildinputs_builds . #nil))) + +(define (handle-build-request db build-id) + "Retrieve build identified by BUILD-ID in DB and convert it to hydra + format. Return #f is not build was found." + (let ((build (db-get-build db build-id))) + (and=> build build->hydra-build))) -(define (object->json-scm obj) - "Prepare OBJ for JSON usage." - (cond ((string? obj) obj) - ((number? obj) obj) - ((boolean? obj) obj) - ((null? obj) obj) - ((symbol? obj) (symbol->string obj)) - ((keyword? obj) (object->json-scm (keyword->symbol obj))) - ((alist? obj) (alist->hash-table (map object->json-scm obj))) - ((pair? obj) (cons (object->json-scm (car obj)) - (object->json-scm (cdr obj)))) - (else (object->string obj)))) - -(define* (spec->json-string spec #:key pretty) - "Return SPEC as a JSON object." - (scm->json-string (object->json-scm spec) #:pretty pretty)) +(define (handle-builds-request db filters) + "Retrieve all builds matched by FILTERS in DB and convert them to hydra + format." + (let ((builds (db-get-builds db filters))) + (map build->hydra-build builds))) + +(define (handle-log-request db build) + "Retrieve the log file of BUILD. Return a lambda which PORT argument is an + input port from which the content of the log file can be read or #f if the + log file is not readable." + (let* ((log (assq-ref build #:log)) + (access (and (string? log) + (access? log R_OK)))) + (and access + (lambda (out-port) + (let ((in-pipe-port + (open-input-pipe + (format #f "~a -dc ~a" %bzip2 log)))) + (dump-port in-pipe-port out-port) + (close-pipe in-pipe-port)))))) + +(define (request-parameters request) + "Parse the REQUEST query parameters and return them under the form + '((parameter value) ...)." + (let* ((uri (request-uri request)) + (query (uri-query uri))) + (and query + (map (lambda (param) + (match (string-split param #\=) + ((key param) + (list (string->symbol key) param)))) + (string-split query #\&))))) ;;; ;;; Web server. ;;; +;;; The api is derived from the hydra one. It is partially described here : +;;; +;;; https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml +;;; (define (request-path-components request) (split-and-decode-uri-path (uri-path (request-uri request)))) (define (url-handler request body db) + (define* (respond response #:key body (db db)) (values response body db)) + + (define-syntax-rule (respond-json body ...) + (respond '((content-type . (application/json))) + #:body body ...)) + + (define-syntax-rule (respond-text body ...) + (respond '((content-type . (text/plain))) + #:body body ...)) + + (define-syntax-rule (respond-json-with-error error-code message) + (respond + (build-response #:headers '((content-type . (application/json))) + #:code error-code) + #:body + (object->json-string + `((error . ,message))))) + + (define (respond-build-not-found build-id) + (respond-json-with-error + 404 + (format #f "Build with ID ~a doesn't exist." build-id))) + + (define (respond-build-log-not-found build) + (let ((drv (assq-ref build #:derivation))) + (respond-json-with-error + 404 + (format #f "The build log of derivation ~a is not available." drv)))) + (match (request-path-components request) (((or "jobsets" "specifications") . rest) - (respond '((content-type . (application/json))) - #:body (spec->json-string (car (db-get-specifications db))))) + (respond-json (object->json-string (car (db-get-specifications db))))) + (("build" build-id) + (let ((hydra-build (handle-build-request db build-id))) + (if hydra-build + (respond-json (object->json-string hydra-build)) + (respond-build-not-found build-id)))) + (("build" build-id "log" "raw") + (let ((build (db-get-build db build-id))) + (if build + (let ((log-response (handle-log-request db build))) + (if log-response + (respond-text log-response) + (respond-build-log-not-found build))) + (respond-build-not-found build-id)))) + (("api" "latestbuilds") + (let* ((params (request-parameters request)) + ;; 'nr parameter is mandatory to limit query size. + (valid-params? (assq-ref params 'nr))) + (if valid-params? + (respond-json (object->json-string + (handle-builds-request db params))) + (respond-json-with-error 500 "Parameter not defined!")))) (_ (respond (build-response #:code 404) #:body (string-append "Resource not found: " @@ -73,6 +169,6 @@ (define* (run-cuirass-server db #:key (port 8080)) (format (current-error-port) "listening on port ~A~%" port) (run-server url-handler - 'http ;server implementation - `(#:port ,port) ;implementation parameters - db)) ;state + 'http + `(#:port ,port) + db)) diff --git a/src/cuirass/utils.scm b/src/cuirass/utils.scm index d966543..a932674 100644 --- a/src/cuirass/utils.scm +++ b/src/cuirass/utils.scm @@ -21,9 +21,29 @@ (define-module (cuirass utils) #:use-module (ice-9 match) #:use-module (srfi srfi-1) - #:export (alist?)) + #:use-module (json) + #:export (alist? + object->json-scm + object->json-string)) (define (alist? obj) "Return #t if OBJ is an alist." (and (list? obj) (every pair? obj))) + +(define (object->json-scm obj) + "Prepare OBJ for JSON usage." + (cond ((string? obj) obj) + ((number? obj) obj) + ((boolean? obj) obj) + ((null? obj) obj) + ((symbol? obj) (symbol->string obj)) + ((keyword? obj) (object->json-scm (keyword->symbol obj))) + ((alist? obj) (map object->json-scm obj)) + ((pair? obj) (cons (object->json-scm (car obj)) + (object->json-scm (cdr obj)))) + (else (object->string obj)))) + +(define* (object->json-string object #:key pretty) + "Return OBJECT as a JSON object." + (scm->json-string (object->json-scm object) #:pretty pretty)) -- 2.13.2 From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 01 15:51:41 2017 Received: (at 27876) by debbugs.gnu.org; 1 Aug 2017 19:51:41 +0000 Received: from localhost ([127.0.0.1]:37306 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcdCe-0001IE-VT for submit@debbugs.gnu.org; Tue, 01 Aug 2017 15:51:41 -0400 Received: from mail-wm0-f44.google.com ([74.125.82.44]:35266) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcdCc-0001Hh-JV for 27876@debbugs.gnu.org; Tue, 01 Aug 2017 15:51:39 -0400 Received: by mail-wm0-f44.google.com with SMTP id m85so24026929wma.0 for <27876@debbugs.gnu.org>; Tue, 01 Aug 2017 12:51:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xYqxAd1Y0VN0CHOZfmWvDdbUcIv/mMHnWHPb95dAFJs=; b=NjFjPEUEzgtREAC6vWy3LFtQtJ5LsaLAADc91OxTMSrBTEdywJ/m6kNIycdvFCPaP/ JmU1z6SZ5Kxwsq5XgQlTKIXeLRRs/kJerKJHeRr6kVKinA9iJYNnYeM4ZFFIoJo+hJGE 7n0QGiftbe5CiNTLm4hOwVlhaP+/TEkLp0ZJJks4vgB398zDz7oG7KzpQTgCQS+Z3AzI cCkmlctM2AVj8dAShU5LXLyN33LAxGc9R3mPBLKGaYnnIzdQJD8K2JzTS5cxAW3fk+/q zVPl3rXBXpNlUtiqcTX+KAngcv6p1phLvtwhBRUnn1weOAZPpqh3WqtSQEOZ/hmXNGlZ FR0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xYqxAd1Y0VN0CHOZfmWvDdbUcIv/mMHnWHPb95dAFJs=; b=EbNjsjnQY0/j8E3D98nATzqaRLlscowJdRsBQO9cNGFoEtBcuejwYtEL64hxClba3F 1YMhfyhcZJGlLUlWcLssKlZeHApM4/ZeKnBHtI0vfaigGC9BRF/wqo9gUCm+6XUrysBI DaoLoMyX1T6//G0jNyiEAzyh1UwSLpbrGHrXTHlin/b/+7MEw+x0DK6U/f3LiWgagPfy YbWAx64w/EMYKMLTmPLn70n6jpTCN7uENrfIbH38oBGLK32q62IxXJHAVS7MaL+ZGAnk zVGfyUubUL2dwdDfB3AEZ+aG+cB3IH7db5ENzhLkZG9yGuRTLBf1irmZe/Q9YTZjHZTd NvZA== X-Gm-Message-State: AIVw110PzDdObC9vwga/x0dHBMA3TKNvK0HjzdVW/pIDaE8stRuSCITo rQfrVi0tc3RoUsjB X-Received: by 10.28.60.8 with SMTP id j8mr2215021wma.166.1501617092853; Tue, 01 Aug 2017 12:51:32 -0700 (PDT) Received: from localhost.localdomain (ABayonne-551-1-53-108.w90-11.abo.wanadoo.fr. [90.11.72.108]) by smtp.gmail.com with ESMTPSA id e76sm2107696wmd.36.2017.08.01.12.51.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 Aug 2017 12:51:32 -0700 (PDT) From: Mathieu Othacehe To: 27876@debbugs.gnu.org Subject: [PATCH v2 3/3] cuirass: Add tests for new HTTP API. Date: Tue, 1 Aug 2017 21:51:24 +0200 Message-Id: <20170801195124.7030-3-m.othacehe@gmail.com> X-Mailer: git-send-email 2.13.2 In-Reply-To: <20170801195124.7030-1-m.othacehe@gmail.com> References: <20170801195124.7030-1-m.othacehe@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 27876 Cc: Mathieu Othacehe X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) * tests/http.scm: Add various tests on new HTTP API. --- tests/http.scm | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 192 insertions(+), 27 deletions(-) diff --git a/tests/http.scm b/tests/http.scm index 4c5214d..99daf23 100644 --- a/tests/http.scm +++ b/tests/http.scm @@ -1,6 +1,7 @@ ;;; http.scm -- tests for (cuirass http) module ;;; Copyright © 2016 Mathieu Lirzin ;;; Copyright © 2017 Ludovic Courtès +;;; Copyright © 2017 Mathieu Othacehe ;;; ;;; This file is part of Cuirass. ;;; @@ -18,7 +19,14 @@ ;;; along with Cuirass. If not, see . (use-modules (cuirass http) + (cuirass database) + (cuirass utils) + (guix utils) + (guix build utils) (json) + (web client) + (web response) + (rnrs bytevectors) (srfi srfi-1) (srfi srfi-64)) @@ -42,30 +50,187 @@ #t t1))) -(test-begin "http") - -(test-assert "spec->json-string" - ;; Note: We cannot compare the strings directly because field ordering - ;; depends on the hash algorithm used in Guile's hash tables, and that - ;; algorithm changed in Guile 2.2. - (hash-table=? - (call-with-input-string - (string-append "{" - "\"boolean\" : false," - "\"string\" : \"guix\"," - "\"alist\" : {\"subset\" : \"hello\"}," - "\"list\" : [1, \"2\", \"three\"]," - "\"symbol\" : \"hydra-jobs\"," - "\"number\" : 1" - "}") - json->scm) - (call-with-input-string - (spec->json-string '((#:number . 1) - (string . "guix") - ("symbol" . hydra-jobs) - (#:alist (subset . "hello")) - (list 1 "2" #:three) - ("boolean" . #f))) - json->scm))) - -(test-end) +(define (http-get-body uri) + (call-with-values (lambda () (http-get uri)) + (lambda (response body) body))) + +(define (wait-until-ready port) + ;; Wait until the server is accepting connections. + (let ((conn (socket PF_INET SOCK_STREAM 0))) + (let loop () + (unless (false-if-exception + (connect conn AF_INET (inet-pton AF_INET "127.0.0.1") port)) + (loop))))) + +(define (test-cuirass-uri route) + (string-append "http://localhost:6688" route)) + +(define database-name + ;; Use an empty and temporary database for the tests. + (string-append (getcwd) "/" (number->string (getpid)) "-tmp.db")) + +(define %db + ;; Global Slot for a database object. + (make-parameter #t)) + +(define build-query-result + '((#:id . 1) + (#:project . "guix") + (#:jobset . "master") + (#:job . "fake-job") + (#:timestamp . 1501347493) + (#:starttime . 1501347493) + (#:stoptime . 1501347493) + (#:buildoutputs . ((out ("path" . "/gnu/store/fake-1.0")))) + (#:system . "x86_64-linux") + (#:nixname . "fake-1.0") + (#:buildstatus . 0) + (#:busy . 0) + (#:priority . 0) + (#:finished . 1) + (#:buildproducts . #nil) + (#:releasename . #nil) + (#:buildinputs_builds . #nil))) + +(define log-file-name + ;; Use a fake temporary log file. + (string-append (getcwd) "/" (number->string (getpid)) "-log.txt")) + +(call-with-output-file log-file-name + ;; Write "build log" string compressed with bzip2 inside LOG-FILE-NAME. + (lambda (out) + (dump-port + (call-with-input-string "build log" + (lambda (port) + (compressed-port 'bzip2 port))) + out))) + +(test-group-with-cleanup "http" + (test-assert "object->json-string" + ;; Note: We cannot compare the strings directly because field ordering + ;; depends on the hash algorithm used in Guile's hash tables, and that + ;; algorithm changed in Guile 2.2. + (hash-table=? + (call-with-input-string + (string-append "{" + "\"boolean\" : false," + "\"string\" : \"guix\"," + "\"alist\" : {\"subset\" : \"hello\"}," + "\"list\" : [1, \"2\", \"three\"]," + "\"symbol\" : \"hydra-jobs\"," + "\"number\" : 1" + "}") + json->scm) + (call-with-input-string + (object->json-string '((#:number . 1) + (string . "guix") + ("symbol" . hydra-jobs) + (#:alist (subset . "hello")) + (list 1 "2" #:three) + ("boolean" . #f))) + json->scm))) + + (test-assert "db-init" + (%db (db-init database-name))) + + (test-assert "cuirass-run" + (call-with-new-thread + (lambda () + (run-cuirass-server (%db) #:port 6688)))) + + (test-assert "wait-server" + (wait-until-ready 6688)) + + (test-assert "fill-db" + (let ((build + `((#:derivation . "/gnu/store/fake.drv") + (#:eval-id . 1) + (#:log . ,log-file-name) + (#:status . 0) + (#:outputs . (("out" . "/gnu/store/fake-1.0"))) + (#:timestamp . 1501347493) + (#:starttime . 1501347493) + (#:stoptime . 1501347493))) + (derivation + '((#:derivation . "/gnu/store/fake.drv") + (#:job-name . "fake-job") + (#:system . "x86_64-linux") + (#:nix-name . "fake-1.0") + (#:eval-id . 1))) + (specification + '((#:name . "guix") + (#:url . "git://git.savannah.gnu.org/guix.git") + (#:load-path . ".") + (#:file . "/tmp/gnu-system.scm") + (#:proc . hydra-jobs) + (#:arguments (subset . "hello")) + (#:branch . "master") + (#:tag . #f) + (#:commit . #f) + (#:no-compile? . #f))) + (evaluation + '((#:specification . "guix") + (#:revision . 1)))) + (db-add-build (%db) build) + (db-add-derivation (%db) derivation) + (db-add-specification (%db) specification) + (db-add-evaluation (%db) evaluation))) + + (test-assert "/build/1" + (hash-table=? + (call-with-input-string + (utf8->string + (http-get-body (test-cuirass-uri "/build/1"))) + json->scm) + (call-with-input-string + (object->json-string build-query-result) + json->scm))) + + (test-equal "/build/1/log/raw" + "build log" + (http-get-body + (test-cuirass-uri "/build/1/log/raw"))) + + (test-equal "/build/2" + 404 + (response-code (http-get (test-cuirass-uri "/build/2")))) + + (test-equal "/build/2/log/raw" + 404 + (response-code (http-get (test-cuirass-uri "/build/2/log/raw")))) + + (test-equal "/api/latestbuilds" + 500 + (response-code (http-get (test-cuirass-uri "/api/latestbuilds")))) + + (test-assert "/api/latestbuilds?nr=1&project=guix&jobset=master" + (let ((hash-list + (call-with-input-string + (utf8->string + (http-get-body + (test-cuirass-uri + "/api/latestbuilds?nr=1&project=guix&jobset=master"))) + json->scm))) + (and (= (length hash-list) 1) + (hash-table=? + (car hash-list) + (call-with-input-string + (object->json-string build-query-result) + json->scm))))) + + (test-assert "/api/latestbuilds?nr=1&project=gnu" + ;; The result should be an empty JSON array. + (let ((hash-list + (call-with-input-string + (utf8->string + (http-get-body + (test-cuirass-uri + "/api/latestbuilds?nr=1&project=gnu"))) + json->scm))) + (= (length hash-list) 0))) + + (test-assert "db-close" + (db-close (%db))) + + (delete-file database-name) + (delete-file log-file-name)) -- 2.13.2 From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 02 05:22:36 2017 Received: (at 27876) by debbugs.gnu.org; 2 Aug 2017 09:22:37 +0000 Received: from localhost ([127.0.0.1]:37660 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcprQ-0004fL-OR for submit@debbugs.gnu.org; Wed, 02 Aug 2017 05:22:36 -0400 Received: from eggs.gnu.org ([208.118.235.92]:49197) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dcprP-0004f8-71 for 27876@debbugs.gnu.org; Wed, 02 Aug 2017 05:22:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dcprF-0002b7-Ud for 27876@debbugs.gnu.org; Wed, 02 Aug 2017 05:22:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:48289) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcprF-0002b0-R9; Wed, 02 Aug 2017 05:22:25 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:39062 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dcprE-0001nZ-3K; Wed, 02 Aug 2017 05:22:24 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH] cuirass: add Hydra compatible HTTP API. References: <20170730100759.17734-1-m.othacehe@gmail.com> <87ini84ptn.fsf@gnu.org> <877eynnk7a.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 Thermidor an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 02 Aug 2017 11:22:21 +0200 In-Reply-To: <877eynnk7a.fsf@gmail.com> (Mathieu Othacehe's message of "Tue, 01 Aug 2017 21:48:09 +0200") Message-ID: <87zibitjci.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876 Cc: 27876@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Heya, Mathieu Othacehe skribis: >> That said, there=E2=80=99s only one place where we work, which is where = we spawn ^^^^ I meant =E2=80=9Cfork=E2=80=9D. :-) >> the =E2=80=98evaluate=E2=80=99 command. It may be that spawning it with= =E2=80=98open-pipe=E2=80=99 >> from (ice-9 popen) would sidestep the problem because =E2=80=98open-pipe= =E2=80=99 is >> specifically written to permit this. > > The open-pipe solution seems fine indeed ! Good. > Yes my planning is the following : > > 1. Fiberizing the whole thing to make it scalable. > 2. Add build start/stop detection : I think the ideal would be wip-ui > branch to get merged before, so that I can use the same mechanism to > parse build output :) > 3. Add authentification, and the other API you're describing. Sounds good! So it looks like you kind of like the =E2=80=98wip-ui=E2=80=99 approach. I= wasn=E2=80=99t sure but maybe that=E2=80=99s the way to go. After all, even if it=E2=80=99s a = bit of a hack, the API itself could still work if/when we have a better protocol with the daemon. We=E2=80=99ll see=E2=80=A6 Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 08 11:59:42 2017 Received: (at 27876) by debbugs.gnu.org; 8 Sep 2017 15:59:42 +0000 Received: from localhost ([127.0.0.1]:56193 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqLh0-0005qC-4z for submit@debbugs.gnu.org; Fri, 08 Sep 2017 11:59:42 -0400 Received: from eggs.gnu.org ([208.118.235.92]:35518) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqLgz-0005py-61 for 27876@debbugs.gnu.org; Fri, 08 Sep 2017 11:59:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dqLgq-0006cH-RE for 27876@debbugs.gnu.org; Fri, 08 Sep 2017 11:59:36 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38199) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dqLgq-0006cD-NV; Fri, 08 Sep 2017 11:59:32 -0400 Received: from [193.50.110.231] (port=41830 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dqLgq-0002EM-Ao; Fri, 08 Sep 2017 11:59:32 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> Date: Fri, 08 Sep 2017 17:59:30 +0200 In-Reply-To: <20170801195124.7030-1-m.othacehe@gmail.com> (Mathieu Othacehe's message of "Tue, 1 Aug 2017 21:51:22 +0200") Message-ID: <87o9ql89lp.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876 Cc: 27876@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hi Mathieu, I=E2=80=99m sorry that this patch series fell through the cracks! Mathieu Othacehe skribis: > * bin/evaluate.in (fill-job): New procedure. > (main): Use it to fill informations (nix-name, system) that will later be > added to database. > * doc/cuirass.texi (Database)[Derivation]: Add system and nix_name fields. > (Database)[Builds]: Add id, status, timestamp, starttime and stoptime > fields. Remove output field. > (Database)[Outputs]: New table describing the build outputs. > * src/cuirass/base.scm (build-packages): Add new fields to build object b= efore > adding it to database. > * src/cuirass/database.scm (db-get-build, db-get-builds): New procedures = to get > a build by id from database and a list of builds using filter parameters > respectively. > * src/schema.sql (Outputs) : New table. > (Derivations): Add system and nix_name columns. > (Builds): Remove output column and add id, status, timestamp, starttime a= nd > stoptime columns. Overall looks good. Just some minor comments and you can push: > +(define (fill-job job eval-id) > + "Given JOB assoc list, add EVAL-ID to it. Also process #:nix-name and > + #:system from derivation stored in JOB." Rather: =E2=80=9CAugment the JOB alist with EVAL-ID and additional information gathered from JOB=E2=80=99s #:derivation.=E2=80=9D > + (let ((drv (read-derivation-from-file > + (assq-ref job #:derivation)))) > + ((compose > + (cut acons #:eval-id eval-id <>) > + (cut acons #:nix-name (derivation-name drv) <>) > + (cut acons #:system (derivation-system drv) <>)) > + job))) Rather: `((#:eval-id . ,eval-id) (#:nix-name . ,=E2=80=A6) =E2=80=A6 ,@job)) [...] > Copyright @copyright{} 2016, 2017 Mathieu Lirzin > +Copyright @copyright{} 2017 Mathieu Othacehe Add @* at the end of the previous line. > @quotation > Permission is granted to copy, distribute and/or modify this document > @@ -312,6 +313,13 @@ This field holds the @code{id} of an evaluation from= the >=20=20 > @item job_name > This text field holds the name of the job. > + > +@item system > +This text field holds the system name of the derivation. > + > +@item nix_name > +This text field holds the name of the derivation. Maybe add: =E2=80=9C---e.g., @code{coreutils-8.28}=E2=80=9D. (Am I right?) Otherwise OK! Of course if would be awesome to have more tests, but let=E2=80=99s not del= ay this patch series further. :-) Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 08 12:00:58 2017 Received: (at 27876) by debbugs.gnu.org; 8 Sep 2017 16:00:58 +0000 Received: from localhost ([127.0.0.1]:56197 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqLiE-0006vl-HJ for submit@debbugs.gnu.org; Fri, 08 Sep 2017 12:00:58 -0400 Received: from eggs.gnu.org ([208.118.235.92]:36202) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqLiD-0006oa-Cm for 27876@debbugs.gnu.org; Fri, 08 Sep 2017 12:00:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dqLi5-0007J0-82 for 27876@debbugs.gnu.org; Fri, 08 Sep 2017 12:00:52 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38296) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dqLi5-0007Is-5M; Fri, 08 Sep 2017 12:00:49 -0400 Received: from [193.50.110.231] (port=41834 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dqLi4-0004cw-P6; Fri, 08 Sep 2017 12:00:49 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH v2 2/3] cuirass: add Hydra compatible HTTP API. In-Reply-To: <20170801195124.7030-2-m.othacehe@gmail.com> (Mathieu Othacehe's message of "Tue, 1 Aug 2017 21:51:23 +0200") References: <20170801195124.7030-1-m.othacehe@gmail.com> <20170801195124.7030-2-m.othacehe@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) Date: Fri, 08 Sep 2017 18:00:47 +0200 Message-ID: <87h8wd89jk.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876 Cc: 27876@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Mathieu Othacehe skribis: > * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API. > * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it > object->json-string. > (object->json-scm): Move it utils.scm. > (handle-*-request): New helpers procedures. > (request-parameters): New procedure to parse a request query. > (url-handler): Add new API's. > * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported > procedures moved from http.scm. All right! From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 08 12:01:46 2017 Received: (at 27876) by debbugs.gnu.org; 8 Sep 2017 16:01:46 +0000 Received: from localhost ([127.0.0.1]:56204 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqLiz-0007b5-Sk for submit@debbugs.gnu.org; Fri, 08 Sep 2017 12:01:46 -0400 Received: from eggs.gnu.org ([208.118.235.92]:36464) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqLiv-0007U4-9n for 27876@debbugs.gnu.org; Fri, 08 Sep 2017 12:01:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dqLin-0007dR-6k for 27876@debbugs.gnu.org; Fri, 08 Sep 2017 12:01:36 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:38314) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dqLin-0007dG-3u; Fri, 08 Sep 2017 12:01:33 -0400 Received: from [193.50.110.231] (port=41836 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dqLil-0005WQ-U5; Fri, 08 Sep 2017 12:01:32 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH v2 3/3] cuirass: Add tests for new HTTP API. References: <20170801195124.7030-1-m.othacehe@gmail.com> <20170801195124.7030-3-m.othacehe@gmail.com> Date: Fri, 08 Sep 2017 18:01:30 +0200 In-Reply-To: <20170801195124.7030-3-m.othacehe@gmail.com> (Mathieu Othacehe's message of "Tue, 1 Aug 2017 21:51:24 +0200") Message-ID: <87d17189id.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876 Cc: 27876@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Mathieu Othacehe skribis: > * tests/http.scm: Add various tests on new HTTP API. Excellent (can even be squashed with the previous patch since they go together.) Thank you! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 08 15:13:23 2017 Received: (at 27876-done) by debbugs.gnu.org; 8 Sep 2017 19:13:23 +0000 Received: from localhost ([127.0.0.1]:56370 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqOiQ-0005wR-Sh for submit@debbugs.gnu.org; Fri, 08 Sep 2017 15:13:23 -0400 Received: from mail-wr0-f173.google.com ([209.85.128.173]:38238) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqOiO-0005wB-Vg for 27876-done@debbugs.gnu.org; Fri, 08 Sep 2017 15:13:21 -0400 Received: by mail-wr0-f173.google.com with SMTP id 108so6150044wra.5 for <27876-done@debbugs.gnu.org>; Fri, 08 Sep 2017 12:13:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:message-id:in-reply-to :date:mime-version:content-transfer-encoding; bh=fhnAoOw49dBrBfzACrUtHbSxU1KAzxsvgAoSs7yWHyQ=; b=ZEPO2nVWBDGnmkg9h6D9+KzxuHSTwvVHtFgUtsguaegVr5U+n2JV0hYAZIfiLLg9sU Rzp4Chk8+YbCDAQ0uPKXE02LY/CNiOmzwjd6NJVU3JN/Aayv3ohY3imx0g34oCvru1xY zjZfSoyNhHiQsmWZLghLh/Odd10jiyFz7lwqgCKQO58sbIP7OTRRvb5QGjejHAE/Yk09 fjY5e82RGBqNOCuXsD8X9WSCeQHuYIDOpwf6iiIqE7E35Xis6R6B0d41c09PjP5Oz3tv Q36zB4m1UTd7874YFAPGZoqJjlKvM8wXuOyUJll35S/n5TjuYIl6rAALYFuuPeb6XdMh Y8Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :message-id:in-reply-to:date:mime-version:content-transfer-encoding; bh=fhnAoOw49dBrBfzACrUtHbSxU1KAzxsvgAoSs7yWHyQ=; b=OyVVtreKurnjIY72uOu77kRYLCwRhVFq5w97cR8bZB2BT6wT+amOOlf5PlXUcN7+al xKpIwpXqWSGoK8kEpc8m3CSapZ/BYzBnC9ANW9n371J/JKb6B7zN0EXRJ1Ads3aPXd81 K3St8hxvX6u7SWndAne4zw4IA60ywy4+/mJ/WHt9BF1+Z81WGPzULFwxnwOMRKuqzI9w ZgzvcR1FEiPcfp1bjxcOFvf/O11TRQRwZJTR5QN9uTyxDxHZhzMgVdfY42XXpucHy2yY QGtqitU5PHhSKjNrTdgJI1MwHqjSfiZGDglHldlYBHUpk0JHSFt094b/xVyOaqrP21VW wdnA== X-Gm-Message-State: AHPjjUhqT2mzHPb4ZX7IR5hXbqcz8ehLozTEQ2kWnt2EGOjTAy7cqU81 40UdZkZKryVVFFy/ X-Google-Smtp-Source: ADKCNb6U1QBx6XV5G0VNtVxp+dEssveJcacZzdcq5vhcTu2fTPd867uoFeE0fHel079kFk9xTrKgPw== X-Received: by 10.223.135.252 with SMTP id c57mr2816724wrc.204.1504897993708; Fri, 08 Sep 2017 12:13:13 -0700 (PDT) Received: from cervin ([78.192.96.126]) by smtp.gmail.com with ESMTPSA id b184sm1646112wmf.13.2017.09.08.12.13.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 08 Sep 2017 12:13:13 -0700 (PDT) References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> <87o9ql89lp.fsf@gnu.org> User-agent: mu4e 0.9.18; emacs 25.2.1 From: Mathieu Othacehe To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. Message-ID: <87pob1m2bk.fsf@gmail.com> In-reply-to: <87o9ql89lp.fsf@gnu.org> Date: Fri, 08 Sep 2017 21:13:09 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 27876-done Cc: 27876-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi Ludo ! > I’m sorry that this patch series fell through the cracks! No problem, thanks for reviewing :) > Maybe add: “---e.g., @code{coreutils-8.28}”. (Am I right?) You are ! I pushed this patch and the two following squashed together, as you suggested. Note that your recent patch on "db-add-build" seems useless now that Builds's primary key is an autoincremented index. I removed most of your patch during my rebase, is that ok for you ? Now this is merged, I'll start working on build start/stop detection next week. Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 08 16:44:29 2017 Received: (at 27876-done) by debbugs.gnu.org; 8 Sep 2017 20:44:29 +0000 Received: from localhost ([127.0.0.1]:56466 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqQ8a-0003R8-Mu for submit@debbugs.gnu.org; Fri, 08 Sep 2017 16:44:28 -0400 Received: from eggs.gnu.org ([208.118.235.92]:54862) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqQ8Z-0003Qw-9c for 27876-done@debbugs.gnu.org; Fri, 08 Sep 2017 16:44:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dqQ8S-0004w3-O2 for 27876-done@debbugs.gnu.org; Fri, 08 Sep 2017 16:44:21 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:46738) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dqQ8S-0004vl-Ki; Fri, 08 Sep 2017 16:44:20 -0400 Received: from [2a01:e0a:1d:7270:6a6c:dc17:fc02:cfda] (port=58190 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dqQ8S-0007rT-6f; Fri, 08 Sep 2017 16:44:20 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> <87o9ql89lp.fsf@gnu.org> <87pob1m2bk.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 22 Fructidor an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Fri, 08 Sep 2017 22:44:17 +0200 In-Reply-To: <87pob1m2bk.fsf@gmail.com> (Mathieu Othacehe's message of "Fri, 08 Sep 2017 21:13:09 +0200") Message-ID: <871sngapjy.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876-done Cc: 27876-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Hi! Mathieu Othacehe skribis: > I pushed this patch and the two following squashed together, as you > suggested. Note that your recent patch on "db-add-build" seems useless > now that Builds's primary key is an autoincremented index. > > I removed most of your patch during my rebase, is that ok for you ? The goal of 72f2b6b77cc4e3d7629bdf34e6daee05398b8de1 was to make sure that, if you add the same build twice, it=E2=80=99ll just work and you won= =E2=80=99t get duplicate entries in the database (although the test only tested that =E2=80=98db-add-build=E2=80=99 itself doesn=E2=80=99t throw an exception.) With an autoincremented index, I suppose we might want to make sure that =E2=80=98db-add-build=E2=80=99 does not add a new entry for a build that=E2= =80=99s already in the table. WDYT? Thanks! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Sep 09 03:48:28 2017 Received: (at 27876-done) by debbugs.gnu.org; 9 Sep 2017 07:48:28 +0000 Received: from localhost ([127.0.0.1]:56797 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqaVA-0003uS-8Y for submit@debbugs.gnu.org; Sat, 09 Sep 2017 03:48:28 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:35087) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dqaV9-0003uG-0L for 27876-done@debbugs.gnu.org; Sat, 09 Sep 2017 03:48:27 -0400 Received: by mail-wr0-f193.google.com with SMTP id n64so2114141wrb.2 for <27876-done@debbugs.gnu.org>; Sat, 09 Sep 2017 00:48:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:message-id:in-reply-to :date:mime-version:content-transfer-encoding; bh=ewvdAMFpSUw7H95DtJtBJQyAIsysDQJgShkSj1ZDO7E=; b=IwTDn2w5P6euk9g56RFM9JGiwVhD8OWd2bnUAVhsxGLKX+PXTWUDcgBGGG1Cigt+6A qpR5VHf42z94xr1Ma/M8BF6c533ysGtcynHx0adg5o4FK2/XslQyFx3b7G8FcI0CcA+E ltztoR/0jgRzsELEv83ypxyirieD8VgH5v0ag/QMGim20LgIfm9eH15nQ90DoPbxOsNX koaOfPwv15KSwgWY+c6GNogz750/QMwcif1Ymn1DvQOhjlvqi48PUVcwPH1AbQzXI0nc 9YeYNFc2y85itHFV0NT96ZcK4iI/vuFBEwKq10qWLOfCCXEbTNX+P12kGmFC7GBtKcGj sA/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :message-id:in-reply-to:date:mime-version:content-transfer-encoding; bh=ewvdAMFpSUw7H95DtJtBJQyAIsysDQJgShkSj1ZDO7E=; b=FbjRzXnXYYAFEPEtPrclhMonmTZ7+i1fGwTSot3Z/JMUN/BAOyKRK9DHL2EXNWsOQn ZAO/QJ73RuCoP0eS+dbUbyCBGc4b8VeuwnUG9Nrud3Jkl0DZ95sflCkP4tdgPwzPLFnb bbQIAYhwg0q83rIxB1LGEWRhiTq1qjYp9dPXpGzZWzpbToPDTJVz3dGWnJ3rk6RfkBBL ggBHNNwlWdPlADGpUa+iI8pkEQM8R2fmQBfksHFA+TJsattGOEYLSoiDUz6nc0zh2Gla F/ac03fcfzWVj3Eehl+d2OhtlsBW+t/tYL1z6pgjZFTFvD446eNki0US7/4qOQ/BVl5c 0MUA== X-Gm-Message-State: AHPjjUiqhE3Q0YBPtE1+7Xl+djWaPw/iFV/1rTdSPRyaQlNuWPE9dvP5 deQcGXFm551AFaua X-Google-Smtp-Source: ADKCNb5mTlgdmML2ojV4lh5vIr4A7rqP/ZMMNXkz2lOqUhaCswDJvRrnUdmDm1eTHpqZc8i96XUMBA== X-Received: by 10.223.155.157 with SMTP id d29mr4443009wrc.24.1504943300818; Sat, 09 Sep 2017 00:48:20 -0700 (PDT) Received: from cervin ([78.192.96.126]) by smtp.gmail.com with ESMTPSA id r14sm4964467wra.97.2017.09.09.00.48.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 09 Sep 2017 00:48:20 -0700 (PDT) References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> <87o9ql89lp.fsf@gnu.org> <87pob1m2bk.fsf@gmail.com> <871sngapjy.fsf@gnu.org> User-agent: mu4e 0.9.18; emacs 25.2.1 From: Mathieu Othacehe To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. Message-ID: <87lglomhxo.fsf@gmail.com> In-reply-to: <871sngapjy.fsf@gnu.org> Date: Sat, 09 Sep 2017 09:48:15 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 27876-done Cc: 27876-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) > With an autoincremented index, I suppose we might want to make sure that > ‘db-add-build’ does not add a new entry for a build that’s already in > the table. WDYT? Yes we might, but looking for a build with the same derivation-evaluation-outputs before each build insertion will be an expensive operation (now that this tuple is no longer the primary key). Mathieu From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 10 09:01:38 2017 Received: (at 27876-done) by debbugs.gnu.org; 10 Sep 2017 13:01:38 +0000 Received: from localhost ([127.0.0.1]:58553 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dr1rm-0006em-4u for submit@debbugs.gnu.org; Sun, 10 Sep 2017 09:01:38 -0400 Received: from eggs.gnu.org ([208.118.235.92]:37544) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dr1rk-0006eY-4q for 27876-done@debbugs.gnu.org; Sun, 10 Sep 2017 09:01:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dr1re-0006hk-5x for 27876-done@debbugs.gnu.org; Sun, 10 Sep 2017 09:01:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:60463) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dr1re-0006he-32; Sun, 10 Sep 2017 09:01:30 -0400 Received: from [2a01:e0a:1d:7270:6a6c:dc17:fc02:cfda] (port=35272 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dr1rd-0004S7-JZ; Sun, 10 Sep 2017 09:01:29 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> <87o9ql89lp.fsf@gnu.org> <87pob1m2bk.fsf@gmail.com> <871sngapjy.fsf@gnu.org> <87lglomhxo.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 24 Fructidor an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Sun, 10 Sep 2017 15:01:27 +0200 In-Reply-To: <87lglomhxo.fsf@gmail.com> (Mathieu Othacehe's message of "Sat, 09 Sep 2017 09:48:15 +0200") Message-ID: <878thm907s.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876-done Cc: 27876-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Heya Mathieu, Mathieu Othacehe skribis: >> With an autoincremented index, I suppose we might want to make sure that >> =E2=80=98db-add-build=E2=80=99 does not add a new entry for a build that= =E2=80=99s already in >> the table. WDYT? > > Yes we might, but looking for a build with the same > derivation-evaluation-outputs before each build insertion will be an > expensive operation (now that this tuple is no longer the primary key). OK, but isn=E2=80=99t there a risk of recording the same set of derivations= at every evaluation? Basically, between two subsequent evaluations, 99% of the derivations are exactly the same (assuming an evaluation is triggered at each git push, and each git push adds/modifies just a handful of packages). If we record new builds for these 99%, even though nothing has changed, then that=E2=80=99s potentially bad, no? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 10 09:27:00 2017 Received: (at 27876-done) by debbugs.gnu.org; 10 Sep 2017 13:27:00 +0000 Received: from localhost ([127.0.0.1]:58586 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dr2GK-0007Ei-CG for submit@debbugs.gnu.org; Sun, 10 Sep 2017 09:27:00 -0400 Received: from mail-wm0-f53.google.com ([74.125.82.53]:44148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dr2GJ-0007EW-C5 for 27876-done@debbugs.gnu.org; Sun, 10 Sep 2017 09:26:59 -0400 Received: by mail-wm0-f53.google.com with SMTP id 189so1833179wmh.1 for <27876-done@debbugs.gnu.org>; Sun, 10 Sep 2017 06:26:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:in-reply-to:date :message-id:mime-version:content-transfer-encoding; bh=xUVnxlRG4LZ3cuCbB9IdmaVYQ1qsl3iEmH0oY1fK1wk=; b=lK06lLuiw4tk+8rzQFeqO7d9b1pQ6tnn0Pa39O3cLHen0F+Vny0QAcjeDllUCnaElS ja9Wv3FZjVyogfQyfTK1e9aFA2d88CHv/YDYYrDUU4b6EKTPYW4dQh8WCmIwstdAGUx6 wFjqAHITxX5h7k2teY9q219ky2sCxTRAMdj/+sNiVPy2qCEqcPE/CyuJufpqHroW1ApB HBKSL3Qo0UO9eQHECXCTNCsUld7BM+1Zz36bqWEphnfVqfnIpahfvrVysQJezgWwLp0k KhQtd6ZH+eggkcwKFqQUNOY4UzB11L2HHp4Q+bgDCFsrPdgQcpj3OQAOtMS/oJqezl7h clSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version:content-transfer-encoding; bh=xUVnxlRG4LZ3cuCbB9IdmaVYQ1qsl3iEmH0oY1fK1wk=; b=Z3SNXw3bCJ3kybckgbWU3p3qkeAKSd/qnLybvqqv4XtWgxYZcGso/QlhKxtp+0vdT+ ZgWxyMDjlGiEOwzAdXwVfp761Bo9x1gfg7Xz8vHyCLoOEXaoAx9l1aXNhmphwXWYnn56 5Q2ez336Pd9UlWGMWMrvG5XShlR+z0BURAK/CNIB1MtANtZDQ8EQguaZjSMQcmqJ7LSN 5481ZI2ixezjlkrswggdwn+kbIBGxtpW1z83HoD2jp0V6I9k8viipnVFU9SjzJRczyR6 p6YSzi6r+Zv7h464cPHySB1yp9uCuW+jKSZyHjovUu3K6DiLic653TbYOwiYaAmiQIh1 /tBQ== X-Gm-Message-State: AHPjjUgVQny5nBV6biYed0NjfjsoVAkAg4xK3bSnKz2zmP5w2zbE08C5 i7CZmE6a7Gbv5Tqam9OncaF0ew== X-Google-Smtp-Source: AOwi7QAncOsl0mMZHKgakrzsycV0kCprFDz7263KrFdccQue5O9RCL2JVVFJE4vYF5ljXvIRXYlFlQ== X-Received: by 10.28.109.77 with SMTP id i74mr6287145wmc.67.1505050013226; Sun, 10 Sep 2017 06:26:53 -0700 (PDT) Received: from cervin ([78.192.96.126]) by smtp.gmail.com with ESMTPSA id c46sm5123363wrg.92.2017.09.10.06.26.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 10 Sep 2017 06:26:52 -0700 (PDT) References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> <87o9ql89lp.fsf@gnu.org> <87pob1m2bk.fsf@gmail.com> <871sngapjy.fsf@gnu.org> <87lglomhxo.fsf@gmail.com> <878thm907s.fsf@gnu.org> User-agent: mu4e 0.9.18; emacs 25.2.1 From: Mathieu Othacehe To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. In-reply-to: <878thm907s.fsf@gnu.org> Date: Sun, 10 Sep 2017 15:26:44 +0200 Message-ID: <87a822vg4r.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 27876-done Cc: 27876-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) Hi Ludo, > Basically, between two subsequent evaluations, 99% of the derivations > are exactly the same (assuming an evaluation is triggered at each git > push, and each git push adds/modifies just a handful of packages). If > we record new builds for these 99%, even though nothing has changed, > then that’s potentially bad, no? Hmm, you're right but the situation is already problematic. On each evaluation, N new entries will appear in Derivations table (because the primary key is on derivation,evaluation tuple). N new entries will also appear in Builds table (both with primary key on id or on derivation,evaluation,output tuple). When build start/stop will be detected, only a small part of N (derivations which were rebuilt), will be added to Builds. Checking for pre-existing builds with the same derivation will become useless unless I'm wrong. So there's still the problem of Derivations table growing from N at each commit. I'm not sure what to do about it. Maybe removing the "evaluation" field, and puting primary key on "derivation" only would be better ? Thanks, Mathieu From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 10 16:39:10 2017 Received: (at 27876-done) by debbugs.gnu.org; 10 Sep 2017 20:39:11 +0000 Received: from localhost ([127.0.0.1]:59768 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dr90Y-0000i0-NM for submit@debbugs.gnu.org; Sun, 10 Sep 2017 16:39:10 -0400 Received: from eggs.gnu.org ([208.118.235.92]:52083) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dr90X-0000ho-VJ for 27876-done@debbugs.gnu.org; Sun, 10 Sep 2017 16:39:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dr90S-0007bh-0X for 27876-done@debbugs.gnu.org; Sun, 10 Sep 2017 16:39:04 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_20,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:40494) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dr90R-0007bV-Sq; Sun, 10 Sep 2017 16:39:03 -0400 Received: from [2a01:e0a:1d:7270:6a6c:dc17:fc02:cfda] (port=35594 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1dr90R-00017p-BN; Sun, 10 Sep 2017 16:39:03 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mathieu Othacehe Subject: Re: [bug#27876] [PATCH v2 1/3] cuirass: Store new information in database to prepare new HTTP API integration. References: <20170730100759.17734-1-m.othacehe@gmail.com> <20170801195124.7030-1-m.othacehe@gmail.com> <87o9ql89lp.fsf@gnu.org> <87pob1m2bk.fsf@gmail.com> <871sngapjy.fsf@gnu.org> <87lglomhxo.fsf@gmail.com> <878thm907s.fsf@gnu.org> <87a822vg4r.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 24 Fructidor an 225 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Sun, 10 Sep 2017 22:38:59 +0200 In-Reply-To: <87a822vg4r.fsf@gmail.com> (Mathieu Othacehe's message of "Sun, 10 Sep 2017 15:26:44 +0200") Message-ID: <87shfu5lwc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 27876-done Cc: 27876-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) Howdy, Mathieu Othacehe skribis: >> Basically, between two subsequent evaluations, 99% of the derivations >> are exactly the same (assuming an evaluation is triggered at each git >> push, and each git push adds/modifies just a handful of packages). If >> we record new builds for these 99%, even though nothing has changed, >> then that=E2=80=99s potentially bad, no? > > Hmm, you're right but the situation is already problematic. > > On each evaluation, N new entries will appear in Derivations table > (because the primary key is on derivation,evaluation tuple). > > N new entries will also appear in Builds table (both with primary key on > id or on derivation,evaluation,output tuple). > > When build start/stop will be detected, only a small part of N > (derivations which were rebuilt), will be added to Builds. Checking for > pre-existing builds with the same derivation will become useless unless > I'm wrong. > > So there's still the problem of Derivations table growing from N at each > commit. I'm not sure what to do about it. Maybe removing the > "evaluation" field, and puting primary key on "derivation" only would be > better ? You=E2=80=99re right, I don=E2=80=99t know off the top of my head. Perhaps= we should check the schemas that Hydra uses, it should be similar. Future work! Ludo=E2=80=99. From unknown Mon Jun 23 23:52:32 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 09 Oct 2017 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator