From unknown Wed Sep 10 01:09:09 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#34228 <34228@debbugs.gnu.org> To: bug#34228 <34228@debbugs.gnu.org> Subject: Status: [PATCH 0/5] Display a progress bar for on-going builds Reply-To: bug#34228 <34228@debbugs.gnu.org> Date: Wed, 10 Sep 2025 08:09:09 +0000 retitle 34228 [PATCH 0/5] Display a progress bar for on-going builds reassign 34228 guix-patches submitter 34228 Ludovic Court=C3=A8s severity 34228 normal tag 34228 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 27 17:16:51 2019 Received: (at submit) by debbugs.gnu.org; 27 Jan 2019 22:16:51 +0000 Received: from localhost ([127.0.0.1]:48031 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsjT-00035q-0i for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:16:51 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57988) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsjR-00035b-7z for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:16:49 -0500 Received: from lists.gnu.org ([209.51.188.17]:42489) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gnsjL-0007Qo-VX for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:16:44 -0500 Received: from eggs.gnu.org ([209.51.188.92]:39934) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gnsjK-00061m-Vg for guix-patches@gnu.org; Sun, 27 Jan 2019 17:16:43 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,RCVD_IN_DNSWL_NONE autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gnshp-0006jR-5R for guix-patches@gnu.org; Sun, 27 Jan 2019 17:15:11 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:56692) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gnshn-0006d0-5M; Sun, 27 Jan 2019 17:15:08 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id D65E1807F; Sun, 27 Jan 2019 23:15:00 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tO0AsQk8qtGy; Sun, 27 Jan 2019 23:15:00 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id BAE15807D; Sun, 27 Jan 2019 23:14:59 +0100 (CET) From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH 0/5] Display a progress bar for on-going builds Date: Sun, 27 Jan 2019 23:14:50 +0100 Message-Id: <20190127221450.25754-1-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 185.233.100.1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello Guix! Some build tools (CMake, Ninja, GHC, etc.) emit progress reports like =E2=80=9C[ 3/42]=E2=80=9D. This patch series is about making use of= them to display a progress bar for on-going builds on --verbosity=3D1. This can be tested for example with =E2=80=98ghc-libmpd=E2=80=99, =E2=80=98= hugin=E2=80=99, =E2=80=98libinput=E2=80=99, or with our =E2=80=98module-import-compiled.d= rv=E2=80=99. With =E2=80=98libinput=E2=80=99 and =E2=80=98hugin=E2=80=99 you get a second p= rogress bar starting anew for the tests, but that=E2=80=99s reasonable I think. Eventually, we could prefix the progress bar by the name of the build phase. The completion percentage is kept in records referenced by the record. Then we simply display it as needed. (guix self) will now also emit appropriate progress reports so that =E2=80=98guix pull -v1=E2=80=99 is able to print progress= bars. (I had initially implemented it with unchanged and arranged to do the regexp-matching dance only upon =E2=80=98-v1=E2=80= =99, but the implementation turned out to be somewhat ugly and it=E2=80=99s probably OK to regexp-exec build output lines in general=E2=80=94we already do it for colorization anyway.) Feedback welcome! Ludo=E2=80=99. Ludovic Court=C3=A8s (5): status: Record more information about builds. status: Keep track of build completion as reported by build tools. status: Print a progress bar for on-going builds when possible. self: Produce progress reports compatible with (guix status). pull: Default to verbosity level 1. guix/scripts/pull.scm | 2 +- guix/self.scm | 18 +++-- guix/status.scm | 157 +++++++++++++++++++++++++++++++++++------- tests/status.scm | 59 ++++++++++++---- 4 files changed, 195 insertions(+), 41 deletions(-) --=20 2.20.1 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 27 17:26:34 2019 Received: (at 34228) by debbugs.gnu.org; 27 Jan 2019 22:26:34 +0000 Received: from localhost ([127.0.0.1]:48040 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnssq-0003NW-0G for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:32 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:53924) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnssn-0003My-Ll for 34228@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:30 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id C10864EF4; Sun, 27 Jan 2019 23:26:28 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fYoGXmHqQcIW; Sun, 27 Jan 2019 23:26:27 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 6644A8087; Sun, 27 Jan 2019 23:26:27 +0100 (CET) From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 34228@debbugs.gnu.org Subject: [PATCH 2/5] status: Keep track of build completion as reported by build tools. Date: Sun, 27 Jan 2019 23:26:17 +0100 Message-Id: <20190127222620.26034-2-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190127222620.26034-1-ludo@gnu.org> References: <20190127222620.26034-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 34228 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) * guix/status.scm ()[completion]: New field. (build): Add #:completion parameter. (%percentage-line-rx, %fraction-line-rx): New variables. (update-build): New procedure. (compute-status): Add 'build-log' case. * tests/status.scm ("compute-status, build completion"): New test. --- guix/status.scm | 62 ++++++++++++++++++++++++++++++++++++++++++++---- tests/status.scm | 31 ++++++++++++++++++++++++ 2 files changed, 89 insertions(+), 4 deletions(-) diff --git a/guix/status.scm b/guix/status.scm index 0a5ff59236..0435d14d6a 100644 --- a/guix/status.scm +++ b/guix/status.scm @@ -101,16 +101,17 @@ ;; On-going or completed build. (define-record-type - (%build derivation id system log-file) + (%build derivation id system log-file completion) build? (derivation build-derivation) ;string (.drv file name) (id build-id) ;#f | integer (system build-system) ;string - (log-file build-log-file)) ;#f | string + (log-file build-log-file) ;#f | string + (completion build-completion)) ;#f | integer (percentage) -(define* (build derivation system #:key id log-file) +(define* (build derivation system #:key id log-file completion) "Return a new build." - (%build derivation id system log-file)) + (%build derivation id system log-file completion)) ;; On-going or completed downloads. Downloads can be stem from substitutes ;; and from "builtin:download" fixed-output derivations. @@ -141,6 +142,57 @@ (lambda (download) (string=? item (download-item download)))) +(define %percentage-line-rx + ;; Things like CMake write lines like "[ 10%] gcc -c …". This regexp + ;; matches them. + (make-regexp "^[[:space:]]*\\[ *([0-9]+)%\\]")) + +(define %fraction-line-rx + ;; The 'compiled-modules' derivations and Ninja produce reports like + ;; "[ 1/32]" at the beginning of each line, while GHC prints "[ 6 of 45]". + ;; This regexp matches these. + (make-regexp "^[[:space:]]*\\[ *([0-9]+) *(/|of) *([0-9]+)\\]")) + +(define (update-build status id line) + "Update STATUS based on LINE, a build output line for ID that might contain +a completion indication." + (define (set-completion b %) + (build (build-derivation b) + (build-system b) + #:id (build-id b) + #:log-file (build-log-file b) + #:completion %)) + + (define (find-build) + (find (lambda (build) + (and (build-id build) + (= (build-id build) id))) + (build-status-building status))) + + (define (update %) + (let ((build (find-build))) + (build-status + (inherit status) + (building (cons (set-completion build %) + (delq build (build-status-building status))))))) + + (cond ((string-any #\nul line) + ;; Don't try to match a regexp here. + status) + ((regexp-exec %percentage-line-rx line) + => + (lambda (match) + (let ((% (string->number (match:substring match 1)))) + (update %)))) + ((regexp-exec %fraction-line-rx line) + => + (lambda (match) + (let ((done (string->number (match:substring match 1))) + (total (string->number (match:substring match 3)))) + (update (* 100. (/ done total)))))) + (else + status))) + (define* (compute-status event status #:key (current-time current-time) @@ -242,6 +294,8 @@ compute a new status based on STATUS." (current-time time-monotonic)) #:transferred transferred) downloads))))) + (('build-log (? integer? pid) line) + (update-build status pid line)) (_ status))) diff --git a/tests/status.scm b/tests/status.scm index e3ea768968..f3afadfcd0 100644 --- a/tests/status.scm +++ b/tests/status.scm @@ -180,4 +180,35 @@ (display "@ build-succeeded bar.drv\n" port) (list first second (get-status)))))) +(test-equal "compute-status, build completion" + (list (build-status + (building (list (build "foo.drv" "x86_64-linux" #:id 121)))) + (build-status + (building (list (build "foo.drv" "x86_64-linux" #:id 121 + #:completion 0.)))) + (build-status + (building (list (build "foo.drv" "x86_64-linux" #:id 121 + #:completion 50.)))) + (build-status + (builds-completed (list (build "foo.drv" "x86_64-linux" #:id 121 + #:completion 100.))))) + (let-values (((port get-status) + (build-event-output-port (lambda (event status) + (compute-status event status + #:current-time + (const 'now)))))) + (display "@ build-started foo.drv - x86_64-linux 121\n" port) + (display "@ build-log 121 6\nHello!" port) + (let ((first (get-status))) + (display "@ build-log 121 20\n[ 0/100] building X\n" port) + (display "@ build-log 121 6\nHello!" port) + (let ((second (get-status))) + (display "@ build-log 121 20\n[50/100] building Y\n" port) + (display "@ build-log 121 6\nHello!" port) + (let ((third (get-status))) + (display "@ build-log 121 21\n[100/100] building Z\n" port) + (display "@ build-log 121 6\nHello!" port) + (display "@ build-succeeded foo.drv\n" port) + (list first second third (get-status))))))) + (test-end "status") -- 2.20.1 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 27 17:26:34 2019 Received: (at 34228) by debbugs.gnu.org; 27 Jan 2019 22:26:34 +0000 Received: from localhost ([127.0.0.1]:48042 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsss-0003Na-9W for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:34 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:53930) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsso-0003N1-36 for 34228@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:30 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 791768087; Sun, 27 Jan 2019 23:26:29 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kuI4xYCvQZOj; Sun, 27 Jan 2019 23:26:28 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id BF0618089; Sun, 27 Jan 2019 23:26:27 +0100 (CET) From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 34228@debbugs.gnu.org Subject: [PATCH 3/5] status: Print a progress bar for on-going builds when possible. Date: Sun, 27 Jan 2019 23:26:18 +0100 Message-Id: <20190127222620.26034-3-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190127222620.26034-1-ludo@gnu.org> References: <20190127222620.26034-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 34228 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) * guix/status.scm (print-build-event)[report-build-progress]: New procedure. [print-log-line]: Add ID parameter. Call 'report-build-progress' when appropriate. Adjust callers. --- guix/status.scm | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/guix/status.scm b/guix/status.scm index 0435d14d6a..e3375816c5 100644 --- a/guix/status.scm +++ b/guix/status.scm @@ -441,14 +441,29 @@ addition to build events." (cut colorize-string <> 'RED 'BOLD) identity)) + (define (report-build-progress %) + (let ((% (min (max % 0) 100))) ;sanitize + (erase-current-line port) + (format port "~3d% " (inexact->exact (round %))) + (display (progress-bar % (- (current-terminal-columns) 5)) + port) + (force-output port))) + (define print-log-line (if print-log? (if colorize? - (lambda (line) + (lambda (id line) (display (colorize-log-line line) port)) - (cut display <> port)) - (lambda (line) - (spin! port)))) + (lambda (id line) + (display line port))) + (lambda (id line) + (match (build-status-building status) + ((build) ;single job + (match (build-completion build) + ((? number? %) (report-build-progress %)) + (_ (spin! port)))) + (_ + (spin! port)))))) (unless print-log? (display "\r" port)) ;erase the spinner @@ -552,7 +567,7 @@ addition to build events." ;; through. (display line port) (force-output port)) - (print-log-line line)) + (print-log-line pid line)) (cond ((string-prefix? "substitute: " line) ;; The daemon prefixes early messages coming with 'guix ;; substitute' with "substitute:". These are useful ("updating @@ -565,7 +580,7 @@ addition to build events." (display (info (string-trim-right line)) port) (newline)) (else - (print-log-line line))))) + (print-log-line pid line))))) (_ event))) -- 2.20.1 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 27 17:26:35 2019 Received: (at 34228) by debbugs.gnu.org; 27 Jan 2019 22:26:35 +0000 Received: from localhost ([127.0.0.1]:48044 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsss-0003Nh-Kw for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:34 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:53934) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsso-0003N2-Ix for 34228@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:30 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 00AF68088; Sun, 27 Jan 2019 23:26:30 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i0izgHqzXfsj; Sun, 27 Jan 2019 23:26:29 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 7D504808D; Sun, 27 Jan 2019 23:26:28 +0100 (CET) From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 34228@debbugs.gnu.org Subject: [PATCH 5/5] pull: Default to verbosity level 1. Date: Sun, 27 Jan 2019 23:26:20 +0100 Message-Id: <20190127222620.26034-5-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190127222620.26034-1-ludo@gnu.org> References: <20190127222620.26034-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 34228 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) * guix/scripts/pull.scm (%default-options): Change 'verbosity to 1. --- guix/scripts/pull.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm index 41c7fb289a..32d55f5357 100644 --- a/guix/scripts/pull.scm +++ b/guix/scripts/pull.scm @@ -69,7 +69,7 @@ (multiplexed-build-output? . #t) (graft? . #t) (debug . 0) - (verbosity . 2))) + (verbosity . 1))) (define (show-help) (display (G_ "Usage: guix pull [OPTION]... -- 2.20.1 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 27 17:26:35 2019 Received: (at 34228) by debbugs.gnu.org; 27 Jan 2019 22:26:35 +0000 Received: from localhost ([127.0.0.1]:48046 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnsst-0003No-1I for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:35 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:53918) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnssn-0003Mx-PU for 34228@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:30 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 8A340808C; Sun, 27 Jan 2019 23:26:28 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jru_qZsrOujG; Sun, 27 Jan 2019 23:26:27 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id EC9114EF4; Sun, 27 Jan 2019 23:26:26 +0100 (CET) From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 34228@debbugs.gnu.org Subject: [PATCH 1/5] status: Record more information about builds. Date: Sun, 27 Jan 2019 23:26:16 +0100 Message-Id: <20190127222620.26034-1-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 34228 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) * guix/status.scm (): New record type. (build, matching-build): New procedures. (compute-status): Adjust to manipulate records instead of derivation file names in 'build-status-builds-completed' and 'build-status-building'. (build-event-output-port)[process-line]: Use 'string-split' to preserve spaces. * tests/status.scm ("compute-status, builds + substitutes") ("compute-status, missing events"): Adjust to expect records. Produce complete "build-started" events. ("compute-status, multiplexed build output"): Likewise, and remove "bar.drv" from 'builds-completed'. --- guix/status.scm | 76 +++++++++++++++++++++++++++++++++++++----------- tests/status.scm | 28 +++++++++--------- 2 files changed, 74 insertions(+), 30 deletions(-) diff --git a/guix/status.scm b/guix/status.scm index 93e119bed1..0a5ff59236 100644 --- a/guix/status.scm +++ b/guix/status.scm @@ -50,6 +50,11 @@ build-status-builds-completed build-status-downloads-completed + build? + build + build-derivation + build-system + download? download download-item @@ -85,15 +90,28 @@ ;; Builds and substitutions performed by the daemon. (define-record-type* build-status make-build-status build-status? - (building build-status-building ;list of drv + (building build-status-building ;list of (default '())) (downloading build-status-downloading ;list of (default '())) - (builds-completed build-status-builds-completed ;list of drv + (builds-completed build-status-builds-completed ;list of (default '())) - (downloads-completed build-status-downloads-completed ;list of store items + (downloads-completed build-status-downloads-completed ;list of (default '()))) +;; On-going or completed build. +(define-record-type + (%build derivation id system log-file) + build? + (derivation build-derivation) ;string (.drv file name) + (id build-id) ;#f | integer + (system build-system) ;string + (log-file build-log-file)) ;#f | string + +(define* (build derivation system #:key id log-file) + "Return a new build." + (%build derivation id system log-file)) + ;; On-going or completed downloads. Downloads can be stem from substitutes ;; and from "builtin:download" fixed-output derivations. (define-record-type @@ -113,6 +131,11 @@ "Return a new download." (%download item uri size start end transferred)) +(define (matching-build drv) + "Return a predicate that matches builds of DRV." + (lambda (build) + (string=? drv (build-derivation build)))) + (define (matching-download item) "Return a predicate that matches downloads of ITEM." (lambda (download) @@ -126,15 +149,29 @@ "Given EVENT, a tuple like (build-started \"/gnu/store/...-foo.drv\" ...), compute a new status based on STATUS." (match event - (('build-started drv _ ...) - (build-status - (inherit status) - (building (cons drv (build-status-building status))))) + (('build-started drv "-" system log-file . rest) + (let ((build (build drv system + #:id (match rest + ((pid . _) (string->number pid)) + (_ #f)) + #:log-file (if (string-null? log-file) + #f + log-file)))) + (build-status + (inherit status) + (building (cons build (build-status-building status)))))) (((or 'build-succeeded 'build-failed) drv _ ...) - (build-status - (inherit status) - (building (delete drv (build-status-building status))) - (builds-completed (cons drv (build-status-builds-completed status))))) + (let ((build (find (matching-build drv) + (build-status-building status)))) + ;; If BUILD is #f, this may be because DRV corresponds to a + ;; fixed-output derivation that is listed as a download. + (if build + (build-status + (inherit status) + (building (delq build (build-status-building status))) + (builds-completed + (cons build (build-status-builds-completed status)))) + status))) ;; Note: Ignore 'substituter-started' and 'substituter-succeeded' because ;; they're not as informative as 'download-started' and @@ -146,10 +183,11 @@ compute a new status based on STATUS." ;; because ITEM is different from DRV's output. (build-status (inherit status) - (building (remove (lambda (drv) - (equal? (false-if-exception - (derivation-path->output-path drv)) - item)) + (building (remove (lambda (build) + (let ((drv (build-derivation build))) + (equal? (false-if-exception + (derivation-path->output-path drv)) + item))) (build-status-building status))) (downloading (cons (download item uri #:size size #:start (current-time time-monotonic)) @@ -394,7 +432,7 @@ addition to build events." (N_ "The following build is still in progress:~%~{ ~a~%~}~%" "The following builds are still in progress:~%~{ ~a~%~}~%" (length ongoing)) - ongoing)))) + (map build-derivation ongoing))))) (('build-failed drv . _) (format port (failure (G_ "build of ~a failed")) drv) (newline port) @@ -570,7 +608,11 @@ The second return value is a thunk to retrieve the current state." (define (process-line line) (cond ((string-prefix? "@ " line) - (match (string-tokenize (string-drop line 2)) + ;; Note: Drop the trailing \n, and use 'string-split' to preserve + ;; spaces (the log file part of 'build-started' events can be the + ;; empty string.) + (match (string-split (string-drop (string-drop-right line 1) 2) + #\space) (("build-log" (= string->number pid) (= string->number len)) (set! %build-output-pid pid) (set! %build-output '()) diff --git a/tests/status.scm b/tests/status.scm index 08a3153218..e3ea768968 100644 --- a/tests/status.scm +++ b/tests/status.scm @@ -36,18 +36,18 @@ (test-equal "compute-status, builds + substitutes" (list (build-status - (building '("foo.drv")) + (building (list (build "foo.drv" "x86_64-linux"))) (downloading (list (download "bar" "http://example.org/bar" #:size 500 #:start 'now)))) (build-status - (building '("foo.drv")) + (building (list (build "foo.drv" "x86_64-linux"))) (downloading (list (download "bar" "http://example.org/bar" #:size 500 #:transferred 42 #:start 'now)))) (build-status - (builds-completed '("foo.drv")) + (builds-completed (list (build "foo.drv" "x86_64-linux"))) (downloads-completed (list (download "bar" "http://example.org/bar" #:size 500 #:transferred 500 @@ -58,7 +58,7 @@ (compute-status event status #:current-time (const 'now)))))) - (display "@ build-started foo.drv\n" port) + (display "@ build-started foo.drv - x86_64-linux \n" port) (display "@ substituter-started bar\n" port) (display "@ download-started bar http://example.org/bar 500\n" port) (display "various\nthings\nget\nwritten\n" port) @@ -76,7 +76,8 @@ (test-equal "compute-status, missing events" (list (build-status - (building '("foo.drv")) + (building (list (build "foo.drv" "x86_64-linux" + #:log-file "foo.log"))) (downloading (list (download "baz" "http://example.org/baz" #:size 500 #:transferred 42 @@ -86,7 +87,8 @@ #:transferred 0 #:start 'now)))) (build-status - (builds-completed '("foo.drv")) + (builds-completed (list (build "foo.drv" "x86_64-linux" + #:log-file "foo.log"))) (downloads-completed (list (download "baz" "http://example.org/baz" #:size 500 #:transferred 500 @@ -103,7 +105,7 @@ (compute-status event status #:current-time (const 'now)))))) - (display "@ build-started foo.drv\n" port) + (display "@ build-started foo.drv - x86_64-linux foo.log\n" port) (display "@ download-started bar http://example.org/bar 999\n" port) (display "various\nthings\nget\nwritten\n" port) (display "@ download-progress baz http://example.org/baz 500 42\n" @@ -136,19 +138,19 @@ (test-equal "compute-status, multiplexed build output" (list (build-status - (building '("foo.drv")) + (building (list (build "foo.drv" "x86_64-linux" #:id 121))) (downloading (list (download "bar" "http://example.org/bar" #:size 999 #:start 'now)))) (build-status - (building '("foo.drv")) + (building (list (build "foo.drv" "x86_64-linux" #:id 121))) (downloading (list (download "bar" "http://example.org/bar" #:size 999 #:transferred 42 #:start 'now)))) (build-status - ;; XXX: Should "bar.drv" be present twice? - (builds-completed '("bar.drv" "foo.drv")) + ;; "bar" is now only listed as a download. + (builds-completed (list (build "foo.drv" "x86_64-linux" #:id 121))) (downloads-completed (list (download "bar" "http://example.org/bar" #:size 999 #:transferred 999 @@ -162,8 +164,8 @@ #:derivation-path->output-path (match-lambda ("bar.drv" "bar"))))))) - (display "@ build-started foo.drv 121\n" port) - (display "@ build-started bar.drv 144\n" port) + (display "@ build-started foo.drv - x86_64-linux 121\n" port) + (display "@ build-started bar.drv - armhf-linux bar.log 144\n" port) (display "@ build-log 121 6\nHello!" port) (display "@ build-log 144 50 @ download-started bar http://example.org/bar 999\n" port) -- 2.20.1 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 27 17:26:39 2019 Received: (at 34228) by debbugs.gnu.org; 27 Jan 2019 22:26:40 +0000 Received: from localhost ([127.0.0.1]:48048 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnssx-0003O4-Kl for submit@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:39 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:53942) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gnssp-0003NF-9T for 34228@debbugs.gnu.org; Sun, 27 Jan 2019 17:26:31 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id C02E48089; Sun, 27 Jan 2019 23:26:30 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c0aP3zjfnBod; Sun, 27 Jan 2019 23:26:28 +0100 (CET) Received: from gnu.org (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 2661A808B; Sun, 27 Jan 2019 23:26:28 +0100 (CET) From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 34228@debbugs.gnu.org Subject: [PATCH 4/5] self: Produce progress reports compatible with (guix status). Date: Sun, 27 Jan 2019 23:26:19 +0100 Message-Id: <20190127222620.26034-4-ludo@gnu.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190127222620.26034-1-ludo@gnu.org> References: <20190127222620.26034-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 34228 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) * guix/self.scm (compiled-modules)[build](report-load) (report-compilation): Write "[M/N]" progress reports. Use line-buffering. --- guix/self.scm | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index d1b8256802..f028bdbfdd 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -856,13 +856,23 @@ containing MODULE-FILES and possibly other files as well." (define (report-load file total completed) (display #\cr) (format #t - "loading...\t~5,1f% of ~d files" ;FIXME: i18n + "[~3@a/~3@a] loading...\t~5,1f% of ~d files" + + ;; Note: Multiply TOTAL by two to account for the + ;; compilation phase that follows. + completed (* total 2) + (* 100. (/ completed total)) total) (force-output)) (define (report-compilation file total completed) (display #\cr) - (format #t "compiling...\t~5,1f% of ~d files" ;FIXME: i18n + (format #t "[~3@a/~3@a] compiling...\t~5,1f% of ~d files" + + ;; Add TOTAL to account for the load phase that came + ;; before. + (+ total completed) (* total 2) + (* 100. (/ completed total)) total) (force-output)) @@ -874,8 +884,8 @@ containing MODULE-FILES and possibly other files as well." #:report-load report-load #:report-compilation report-compilation))) - (setvbuf (current-output-port) 'none) - (setvbuf (current-error-port) 'none) + (setvbuf (current-output-port) 'line) + (setvbuf (current-error-port) 'line) (set! %load-path (cons #+module-tree %load-path)) (set! %load-path -- 2.20.1 From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 29 06:11:08 2019 Received: (at 34228-done) by debbugs.gnu.org; 29 Jan 2019 11:11:08 +0000 Received: from localhost ([127.0.0.1]:49738 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1goRIK-0001Fi-K0 for submit@debbugs.gnu.org; Tue, 29 Jan 2019 06:11:08 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:53448) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1goRII-0001Fa-OF for 34228-done@debbugs.gnu.org; Tue, 29 Jan 2019 06:11:07 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id ED7508B31 for <34228-done@debbugs.gnu.org>; Tue, 29 Jan 2019 12:11:05 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v589qoNyMxZt for <34228-done@debbugs.gnu.org>; Tue, 29 Jan 2019 12:11:05 +0100 (CET) Received: from ribbon (unknown [IPv6:2001:660:6102:320:e120:2c8f:8909:cdfe]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 46D9A8B17 for <34228-done@debbugs.gnu.org>; Tue, 29 Jan 2019 12:11:05 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 34228-done@debbugs.gnu.org Subject: Re: [bug#34228] [PATCH 0/5] Display a progress bar for on-going builds References: <20190127221450.25754-1-ludo@gnu.org> Date: Tue, 29 Jan 2019 12:11:04 +0100 In-Reply-To: <20190127221450.25754-1-ludo@gnu.org> ("Ludovic \=\?utf-8\?Q\?Cou\?\= \=\?utf-8\?Q\?rt\=C3\=A8s\=22's\?\= message of "Sun, 27 Jan 2019 23:14:50 +0100") Message-ID: <87k1inoh13.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 34228-done X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ludovic Court=C3=A8s skribis: > Some build tools (CMake, Ninja, GHC, etc.) emit progress reports > like =E2=80=9C[ 3/42]=E2=80=9D. This patch series is about making use of= them > to display a progress bar for on-going builds on --verbosity=3D1. Pushed! 2790b6670b pull: Default to verbosity level 1. 35dcaa119e self: Produce progress reports compatible with (guix status). 3854c6429c status: Print a progress bar for on-going builds when possible. 73a8681a16 status: Keep track of build completion as reported by build to= ols. 976ef2d978 status: Record more information about builds. Ludo'. From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 29 06:50:30 2019 Received: (at 34228-done) by debbugs.gnu.org; 29 Jan 2019 11:50:31 +0000 Received: from localhost ([127.0.0.1]:49775 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1goRuQ-0002K5-LG for submit@debbugs.gnu.org; Tue, 29 Jan 2019 06:50:30 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21042) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1goRuN-0002Ju-Fx for 34228-done@debbugs.gnu.org; Tue, 29 Jan 2019 06:50:29 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1548762590; cv=none; d=zoho.com; s=zohoarc; b=CSeZ9ieS8UE8LELlFKXJAfJGPaNrfxvn39Ev7tTi5oydpWHlKz/G4jqX3U5dQ3IiSrBFRe7PbqCjMBC4sbpAeVWyTv36tE2pH9/3ynpAL6UrHdFqhyL2Bd33qoM8337ipjstdqW+7mwgRKhYyrpHf7kX4bsks6kdCbtDChi+JjI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1548762590; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=Zf44w72FvVe8JRiG7wZG/a4dYxkKZWsFqMAQkdcEha0=; b=Bhljk8vRxEE1H46VPnWINANM8ZMWYhTs3G4ySXxHuzVWtzTKkSr4GHx1+28APCqkUS1WGzR79UqCNXRFn/KYDD8uy54QfVYLxGFDWqW/jNKYb4EW0yvsT1LCr1Zvx/8qn+TwEE0+D/1g0knZz564by4FmD6GcDHve/8Jx5U1p0g= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1548762590; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; l=739; bh=Zf44w72FvVe8JRiG7wZG/a4dYxkKZWsFqMAQkdcEha0=; b=TMUr+g+82SO7Z/KQe/c+QwAlxsmDtdwPYhCPFU8rqN8ReMVzYkfw6OVHopahXY4a C8X2vxDOtpUMT6yvaLRLUi+XfBVFggfflP56ZhgjZmDwyIF9FHnbdyUtcE8jY8zYrqw UcU03IVxcJl8GEDHuTTQRJjMjlD1+1yAtFJ3gFM4= Received: from localhost (141.80.247.225 [141.80.247.225]) by mx.zohomail.com with SMTPS id 1548762589957133.55275654146703; Tue, 29 Jan 2019 03:49:49 -0800 (PST) References: <20190127221450.25754-1-ludo@gnu.org> <87k1inoh13.fsf@gnu.org> User-agent: mu4e 1.0; emacs 26.1 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#34228: [PATCH 0/5] Display a progress bar for on-going builds In-reply-to: <87k1inoh13.fsf@gnu.org> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 29 Jan 2019 12:49:45 +0100 Message-ID: <87tvhrptt2.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 34228-done Cc: 34228-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ludovic Court=C3=A8s writes: > Ludovic Court=C3=A8s skribis: > >> Some build tools (CMake, Ninja, GHC, etc.) emit progress reports >> like =E2=80=9C[ 3/42]=E2=80=9D. This patch series is about making use o= f them >> to display a progress bar for on-going builds on --verbosity=3D1. > > Pushed! > > 2790b6670b pull: Default to verbosity level 1. > 35dcaa119e self: Produce progress reports compatible with (guix status). > 3854c6429c status: Print a progress bar for on-going builds when possib= le. > 73a8681a16 status: Keep track of build completion as reported by build = tools. > 976ef2d978 status: Record more information about builds. This is great! Thank you! --=20 Ricardo From unknown Wed Sep 10 01:09:09 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 26 Feb 2019 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator