From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:19:09 2021 Received: (at submit) by debbugs.gnu.org; 18 Jan 2021 06:19:09 +0000 Received: from localhost ([127.0.0.1]:46132 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nsb-0003hQ-11 for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:19:09 -0500 Received: from lists.gnu.org ([209.51.188.17]:46224) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1NsZ-0003hI-2b for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:19:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:58622) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l1NsY-00062C-RP for guix-patches@gnu.org; Mon, 18 Jan 2021 01:19:06 -0500 Received: from mail-qt1-x834.google.com ([2607:f8b0:4864:20::834]:41775) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l1NsX-0006R8-84 for guix-patches@gnu.org; Mon, 18 Jan 2021 01:19:06 -0500 Received: by mail-qt1-x834.google.com with SMTP id j26so10573077qtq.8 for ; Sun, 17 Jan 2021 22:19:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=w6R3H6ya07xCT/lPgoNUIq5kvfJCTOsgk8wn5XLvWYM=; b=sjZbS8i5cywqr7yQAjTqemu6TNuYP4CLO1qNrcl4sl6ulbdxLX6rxvwzmOPIa+ECKK tfMVqCzCOJ8GXX/SBDKLX7Onapd44YMFOzgTUoXgi94XRwwO8dgBAHpdI3HNnaG/Uof/ VXRzAIdU2vRJCmAevngDRHuN9BcQ2bIbc0MACb32hQhT0ON4FHe260EaQfroFmyLpueA Uzv1rzNbtV86rebmcRTxdbD3Wvb79FSFHElSnxmfjkE74ujgT2snANEO/Ixu8XTQx8Y1 yPuQLOS0Cm5Za39V0tEynzu14DmQB4qlkaz9XvSLsG3r8fIpgXbEFRXSlvQf31xIrTEw WgTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=w6R3H6ya07xCT/lPgoNUIq5kvfJCTOsgk8wn5XLvWYM=; b=iEHKDS0USPtTTKRPk8sj5/PyOgshUCLvSEwjDglnihK+DOnTgRUCF911RVVtddB+i+ 1AGbnBt10nosY8Y6boN/UdAmh0aWoEPYawjm5PHPec/zycL+QfD8WcMcmUEaVdjkYYa0 MqYFDN0n+lKsg4QhygIBEjUHsf86efUhSzGIXgPlvEyj9Y6wiOTuiyZ/TGu5qIy41Srr 9pIEfY2S+vKMvhD6zo1OXRsBnsxsCU/so6ymea8GFuklBh4pS1+/QZngigyQGUqqZqVv qu7WMbe726ztDUNkztk7QFUiO4SuUt3kzONOF2ML/KM7W5FUjkY59v3zyptgbz9MRUo+ xwog== X-Gm-Message-State: AOAM5328KZ5sYg613AdBTiH2wwi7aEbTKMrFoUfb0DQEm4QwQ1mj20I2 qDN+GiJnDv2l9Dp4jZohIXaHLqHim0M= X-Google-Smtp-Source: ABdhPJwr3zoduvPMaloN23RgGA8z9iPtlrUGD3cBHaRMVLTy9yt5XzyNxFRZMrwL70OUr8KcQt0uKQ== X-Received: by 2002:ac8:48d9:: with SMTP id l25mr16132235qtr.60.1610950743447; Sun, 17 Jan 2021 22:19:03 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id 60sm9519378qth.14.2021.01.17.22.19.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:19:02 -0800 (PST) From: Maxim Cournoyer To: guix-patches@gnu.org Subject: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. Date: Mon, 18 Jan 2021 01:18:53 -0500 Message-Id: <20210118061853.26808-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::834; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x834.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This series make the SRFI 64 test driver easier to hack on, and add support to select individual test cases (rather than test files) as well as a way to make the output concise enough to make it more useful in a test driven development context. Maxim Cournoyer (5): build: test-driver.scm Make output redirection optional. build: test-driver.scm: Define the --test-name option as required. build: test-driver.scm: Enable colored test results by default. build: test-driver.scm: Add test cases filtering options. build: test-driver.scm: Add a new '--errors-only' option. build-aux/test-driver.scm | 144 +++++++++++++++++++++++++++----------- doc/guix.texi | 24 +++++++ 2 files changed, 127 insertions(+), 41 deletions(-) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:25:22 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:22 +0000 Received: from localhost ([127.0.0.1]:46142 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyc-0003rF-2O for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:22 -0500 Received: from mail-qk1-f171.google.com ([209.85.222.171]:37504) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nya-0003r0-Iq for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:20 -0500 Received: by mail-qk1-f171.google.com with SMTP id h4so17820489qkk.4 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=4D8waMyo463f7J5zM1MjaZklpqHMDdYq3uncGDMN4Bk=; b=HvNmQVAUNsrr9ZzTroMvR6bCJpbgKf5271bOuW6gkZ8JvoZAVWr3+U+yYhLIJnlCQj D3MSmnl4kEJnp/LhtmuVdlZfCdrI6pMj4QKzscwSvXXhDBrJQeeKEq0eR7g5VbWDg/0s xnTOQTMZNjCX1ne16YsGinTMstgPo4r33AMW3YMLe8aVLWXSAZgacq7fqjCRlUxfdcu6 l6rFz1Z2wEGla4cTYlIDAWIUIEuq93sXX9X9Q101RMVaW0Ro84tre1bPLIThpUnTd4xb Mm23PfYWKuyUzsPxB2LHUpdXHW8iiw1G4+FBDHuQdGK2FqZgo9Hy0Z1NO3I5i7Neohta t31g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=4D8waMyo463f7J5zM1MjaZklpqHMDdYq3uncGDMN4Bk=; b=jpbEWHfQSbMjk185FhDxfYUSNrsAQCdA2Boq6Yw6Olz37sx7Ytf26T1IWFVITTZzOE WGeDXePdiaOEc2U35fUXAHzIdBGBWs/FhICmSJM0agB3gtPmMHw4071Vs09ewAxz7fbD ROfQOsuCtOHZhf7gRSjD7hDGFAF5XLh7J3g4P0wHpCoDSHqx0dFYHlfUHsJtDYZ7b1Jy TYTsgqg+2gApCVs0tq2yNwxw/X8MRasQ8FvLt+w7ylK3Hz5sGvKWXazr7oFxE9Lz9Q9G NcinksDi7snJPzTAIzRQ8aBub4NE97wRwOYnWAjPkfq6aAmfWYU678mUqmiLyAen5aG9 Vv6g== X-Gm-Message-State: AOAM532SI/qwVuoKc9xQOP9aw21oygozq6qR4i1M1Z3cr4ZE0jpR962Z tvX+E3tJCZMDlWT9gJlD3DyvYG0qSY8= X-Google-Smtp-Source: ABdhPJwnAxWqy1hISecwYLj42Ny3jQXIeA2szCUNCnBHvY5RguSambN//01UCApOdBSk4pYq7ovlcQ== X-Received: by 2002:a37:52c1:: with SMTP id g184mr23271611qkb.364.1610951114832; Sun, 17 Jan 2021 22:25:14 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:14 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 1/5] build: test-driver.scm Make output redirection optional. Date: Mon, 18 Jan 2021 01:24:56 -0500 Message-Id: <20210118062501.27022-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) This makes it easier (and less surprising) for users to experiment with the custom Scheme test driver directly. The behavior is unchanged from Automake's point of view. * build-aux/test-driver.scm (main): Make the --log-file and --trs-file arguments optional. Only open, redirect and close a port to a log file when the --log-file option is provided. Only open and close a port to a trs file when the --trs-file option is provided. (test-runner-gnu): Set OUT-PORT parameter default value to the current output port. Set the TRS-PORT parameter default value to a void port. Update doc. --- build-aux/test-driver.scm | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 52af1e9be7..b7622c3ed2 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -1,8 +1,9 @@ ;;;; test-driver.scm - Guile test driver for Automake testsuite harness -(define script-version "2017-03-22.13") ;UTC +(define script-version "2021-01-18.06") ;UTC ;;; Copyright © 2015, 2016 Mathieu Lirzin +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This program is free software; you can redistribute it and/or modify it ;;; under the terms of the GNU General Public License as published by @@ -75,11 +76,14 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) "") ;no color result))) -(define* (test-runner-gnu test-name #:key color? brief? out-port trs-port) +(define* (test-runner-gnu test-name #:key color? brief? + (out-port (current-output-port)) + (trs-port (%make-void-port "w"))) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the file name of the current the test. COLOR? specifies whether to use colors, -and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. The -current output port is supposed to be redirected to a '.log' file." +and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. +OUT-PORT defaults to the current output port, while TRS-PORT defaults to a +void port, which means no TRS output is logged." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the @@ -156,20 +160,22 @@ current output port is supposed to be redirected to a '.log' file." ((option 'help #f) (show-help)) ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) (else - (let ((log (open-file (option 'log-file "") "w0")) - (trs (open-file (option 'trs-file "") "wl")) - (out (duplicate-port (current-output-port) "wl"))) - (redirect-port log (current-output-port)) - (redirect-port log (current-warning-port)) - (redirect-port log (current-error-port)) + (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) + (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) + (out (duplicate-port (current-output-port) "wl")) + (test-name (option 'test-name #f))) + (when log + (redirect-port log (current-output-port)) + (redirect-port log (current-warning-port)) + (redirect-port log (current-error-port))) (test-with-runner - (test-runner-gnu (option 'test-name #f) + (test-runner-gnu test-name #:color? (option->boolean opts 'color-tests) #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) - (load-from-path (option 'test-name #f))) - (close-port log) - (close-port trs) + (load-from-path test-name)) + (and=> log close-port) + (and=> trs close-port) (close-port out)))) (exit 0))) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:25:29 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:29 +0000 Received: from localhost ([127.0.0.1]:46146 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyj-0003rd-FM for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:29 -0500 Received: from mail-qk1-f175.google.com ([209.85.222.175]:37515) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyh-0003rG-FO for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:28 -0500 Received: by mail-qk1-f175.google.com with SMTP id h4so17820740qkk.4 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BTN52cGaaJ7LuN3IMX6UbScTvxrkYyA7iVE51RWVBbk=; b=fv7kWLTRH0NwPPn6KOo+7LxmbxWKwhgDMbjv4UAtfflns+LJ4Y2um2CB6UbCoVAYNA wcVbCahzHfRfcybjVik+KIKPWMQBWQ5uhLaz6fbtV1zx+siCICJbkoYcIntq48AGpi0w 3dvv+NECihwGy/jclYGsvvl+I91la0vnjULD1G1FhhCmBleps+4NbM5fLbzTNyZTx20D azS08Kds/wV/irrL/1Di3MGlms7MtivLJn5hKg7rTD3sXem4JWt9Reou/+HvsoYFedrO ejW6zdF8xR9V4nUEo67mkGd78s/M34GyxaAtMSASpRK2WCOrxIAH0jrGkfqFLobrJsy0 t9kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BTN52cGaaJ7LuN3IMX6UbScTvxrkYyA7iVE51RWVBbk=; b=Kjv2uftiMzJKOdWXf4hmVNREZq4CrSCjrUQVNoHRxyclN85G1H5uZqL6+wQwJewhlM cWuqLFZbBA8fnEx7zj3TXmPm1Sbup7MkZ98xEPJR6CS839TbD1+GTXKQMdfxsn3+JLZT v1sC0DjuzehDQctwIEyx3CUmGDmesXfKvasHn6GyF/1RXyvolCTaBUQjhptpEYa08Rpq XQ5FdxYmniIQdWwWLAB6JVTOpVhhlYs3NHWpG9+vsQ/epmjO/Kxw/drtZKivHhdSJB+r y6JrB6GvwGW/tFVkRAHO1LYWg7EgCmtamtuS6/yWhNTIFdTleZZOZzwYPmKyjkeZkpD0 P+OA== X-Gm-Message-State: AOAM533N1PtNtCa3kjJvZFIOPEa1t9gZ5ASKw5soExGrT6jYkkonLHpJ HhoHyf7tdVgrIazH0XlZAjFT6SSGKus= X-Google-Smtp-Source: ABdhPJwRJwPMvQnYddQonRUO3MM/Lefwj3hwsPe7jxxW982fGX0m7pdvli0ZFJJ0HrqA/VtJxwJrvg== X-Received: by 2002:ae9:e50b:: with SMTP id w11mr22740993qkf.306.1610951121941; Sun, 17 Jan 2021 22:25:21 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:21 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 2/5] build: test-driver.scm: Define the --test-name option as required. Date: Mon, 18 Jan 2021 01:24:57 -0500 Message-Id: <20210118062501.27022-2-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) This is clearer than failing to open an empty string file name, when the option is not provided. * build-aux/test-driver.scm (%options)[test-name]: Set the required? property to #t. --- build-aux/test-driver.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index b7622c3ed2..5891631da6 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -39,7 +39,7 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) (define %options - '((test-name (value #t)) + '((test-name (value #t) (required? #t)) (log-file (value #t)) (trs-file (value #t)) (color-tests (value #t)) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:25:33 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:33 +0000 Received: from localhost ([127.0.0.1]:46151 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nym-0003s8-Pv for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:33 -0500 Received: from mail-qv1-f43.google.com ([209.85.219.43]:46115) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyj-0003rQ-HL for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:30 -0500 Received: by mail-qv1-f43.google.com with SMTP id a1so7027051qvd.13 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=gQWH98SyRFPA7QVmOdsO2pB0Vhy2uUM4zEg8CJPOCY4HVYXO03GAlHdft3rq0JiDlP tmQcyO7nepja6gtox7VqvQDgV7p3uGTXvC0ncyZIhucwHwTnQX4kdmTjJNaC6pT3wTlm 3bCpvQHgwfXclSOfFVeuD+e0w/8wFV9/ya0cf/SWSb3BrqLJCnSPR7l4pt2WJXnwBbys hg1H5rW+beX7YB8E4juxkG1b85AlQSF1HFZa6Z7VnA9hT0MC0LP3ZwzkmNfzepwrzava 6jK2mwhuzjcrzJgpwkeq6udgVPgYxuRLnvjOpz9jpgXxblsl9DMXQWNpE2O3a7VgZ3Rs stNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=aQ8aSb93+SExdCdJSRF4s+yb0exEf0nNkGdavKGhPiJ94b5yAszKm3QPWkgY6KX9bn SXsD3Cdadlz0x6EiAMjJg4MX2vRpmg91NSo/EyfI/EB4wYgqc4JNg+8MXcvQvNY9wFNa t5Z/4KlXpI4Phu1Gem15LfHUdVGXOtIRcdoEkdiY0H9dH71vLALXYIdDTuEyedVMJfwn 57o9ffoAIYp0ujRQqxWnN0EVimt1jEZkfeD513uKxNuhbLz+o38qraN5UJTRa/QAdzca eOFjqx5qPvlePnNGaFa1luczqh3+6hRs3AA8AdtaG/knVJYlQZD3q5XPaDatgzcvItdc SuSA== X-Gm-Message-State: AOAM533yXfOUAWh5TQHNcVldFv3ik/4yiqM9B9Z7PQnmIqqEmgR/D2TK 5utcMUtDwoqHEtMelYw12xu01BUHXXk= X-Google-Smtp-Source: ABdhPJyv1oC/q+bCdriupaaNGoaeOfGZQwyaxDAH+YmZCvEjI88wS3tz2rZoMuPkJo47BaNdPiIsfQ== X-Received: by 2002:a05:6214:1583:: with SMTP id m3mr22847389qvw.48.1610951123645; Sun, 17 Jan 2021 22:25:23 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:23 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 4/5] build: test-driver.scm: Add test cases filtering options. Date: Mon, 18 Jan 2021 01:24:59 -0500 Message-Id: <20210118062501.27022-4-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * build-aux/test-driver.scm (show-help): Add help text for the new --select and --exclude options. (%options): Add the new select and exclude options. (test-runner-gnu): Pass them to the test runner. Update doc. (test-match-name*, test-match-name*/negated, %test-match-all): New variables. (main): Compute the test specifier based on the values of the new options and apply it to the current test runner when running the test file. * doc/guix.texi (Running the Test Suite): Document the new options. --- build-aux/test-driver.scm | 66 ++++++++++++++++++++++++++++++++------- doc/guix.texi | 12 +++++++ 2 files changed, 66 insertions(+), 12 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 767a9fb25d..3ad6b0c93f 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -27,6 +27,8 @@ (use-modules (ice-9 getopt-long) (ice-9 pretty-print) + (ice-9 regex) + (srfi srfi-1) (srfi srfi-26) (srfi srfi-64)) @@ -34,14 +36,19 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] + [--select=REGEXP] [--exclude=REGEXP] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] -The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) +The '--test-name', '--log-file' and '--trs-file' options are mandatory. +The '--select' and '--exclude' options allow selecting or excluding individual +test cases via a regexp, respectively.\n")) (define %options '((test-name (value #t) (required? #t)) (log-file (value #t)) (trs-file (value #t)) + (select (value #t)) + (exclude (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -76,14 +83,22 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) "") ;no color result))) + +;;; +;;; SRFI 64 custom test runner. +;;; + (define* (test-runner-gnu test-name #:key color? brief? (out-port (current-output-port)) - (trs-port (%make-void-port "w"))) + (trs-port (%make-void-port "w")) + select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the file name of the current the test. COLOR? specifies whether to use colors, and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged." +void port, which means no TRS output is logged. SELECT and EXCLUDE may take a +regular expression to select or exclude individual test cases based on their +names." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the @@ -148,6 +163,24 @@ void port, which means no TRS output is logged." (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) runner)) + +;;; +;;; SRFI 64 test specifiers. +;;; +(define (test-match-name* regexp) + "Return a test specifier that matches a test name against REGEXP." + (lambda (runner) + (string-match regexp (test-runner-test-name runner)))) + +(define (test-match-name*/negated regexp) + "Return a negated test specifier version of test-match-name*." + (lambda (runner) + (not (string-match regexp (test-runner-test-name runner))))) + +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list +;;; of test specifiers computed at run time. +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) + ;;; ;;; Entry point. @@ -158,15 +191,22 @@ void port, which means no TRS output is logged." (option (cut option-ref opts <> <>))) (cond ((option 'help #f) (show-help)) - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) + ((option 'version #f) (format #t "test-driver.scm ~A" 'script-version)) (else - (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) - (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) - (out (duplicate-port (current-output-port) "wl")) - (test-name (option 'test-name #f)) - (color-tests (if (assoc 'color-tests opts) - (option->boolean opts 'color-tests) - #t))) + (let* ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) + (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) + (out (duplicate-port (current-output-port) "wl")) + (test-name (option 'test-name #f)) + (select (option 'select #f)) + (exclude (option 'exclude #f)) + (test-specifiers (filter-map + identity + (list (and=> select test-match-name*) + (and=> exclude test-match-name*/negated)))) + (test-specifier (apply %test-match-all test-specifiers)) + (color-tests (if (assoc 'color-tests opts) + (option->boolean opts 'color-tests) + #t))) (when log (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) @@ -176,7 +216,9 @@ void port, which means no TRS output is logged." #:color? color-tests #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) - (load-from-path test-name)) + (test-apply test-specifier + (lambda _ + (load-from-path test-name)))) (and=> log close-port) (and=> trs close-port) (close-port out)))) diff --git a/doc/guix.texi b/doc/guix.texi index 194bb3a314..d59d2806cb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -915,6 +915,18 @@ the @code{SCM_LOG_DRIVER_FLAGS} makefile variable as in this example: make check TESTS="tests/base64.scm" SCM_LOG_DRIVER_FLAGS="--brief=no" @end example +The underlying SRFI 64 custom Automake test driver used for the 'check' +test suite (located at @file{build-aux/test-driver.scm}) also allows +selecting which test cases to run at a finer level, via its +@option{--select} and @option{--exclude} options. Here's an example, to +run all the test cases from the @file{tests/packages.scm} test file +whose names start with ``transaction-upgrade-entry'': + +@example +export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" +make check TESTS="tests/packages.scm" +@end example + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:25:33 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:33 +0000 Received: from localhost ([127.0.0.1]:46153 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyn-0003sB-8C for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:33 -0500 Received: from mail-qv1-f53.google.com ([209.85.219.53]:41447) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyi-0003rO-FT for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:31 -0500 Received: by mail-qv1-f53.google.com with SMTP id h16so7038288qvu.8 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kfLiN+P6E3P5dDlIzdp5yUV5X6s1u1cWIy9n5+yzMpc=; b=YgX2cEdryyv8FsxLR8VjkKYfHP6yDnoPmFrumW1Dckuk48vY6QSs5EtHSLf9NSZReP Zm/8uiHP7EmnVfFS2n606aA+iJ+jfzq/7Wi9oxrVbs7fIoxacOb6sPc5Z3Vwz/lPg/72 6KpI6qnDJlBebDYhoPRIqP1Elx5YzV8xDg12zq04ooZXgrIF86HO287Opy0TYVh3BevV Ax2taqUl0ma46yEZEhbtWpw7ovz70+kcuanGh8QCPbuVFApu2OBuS77Ku5JYVXz9lNsb hT0q1Pka7PUnFDwWJCUNYrZMg/FrJNfb891Y5A71RI+Kz4RGeNnlVqepJTr10MWCsJy2 cOag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kfLiN+P6E3P5dDlIzdp5yUV5X6s1u1cWIy9n5+yzMpc=; b=mOUje191r2REn4E/yKfQgmJ9rXu2t/QOCCu+5rfd/WNQkGOYNMsCf+f2EctjIdvxzX U9E/i0dJijiPnN20ePXw/91AeG8uJEVW5jvYAIPVbGagrC0rwh7umL0piWe8tgmsX9cH 57i4eGurI4suTeEnhJOhrxbd7CZJ/s5vLX20b/QFD+yNiNrtV4MvmWM/nZeMiCQ5O/2Z eqMkFH86XGv46kjTuaK3AT+/Phx3sAt5PiAHwA97IZvWNoPVKCSyr4j/0QnpoVRG3w7J +GO1hgdZMKi4PO7Csf6nUz6b9sdOCWMeuKdrACplD7aqTB3udxyzms3FUt/XUlPINNij 9/pg== X-Gm-Message-State: AOAM532X7PKt+GKl65cSa3O7YTqUztog2WV2gMuprSSRu2g6TG55Qi/X SpL3M32wtx0hSNgzPwFSW6p2TgO9tiM= X-Google-Smtp-Source: ABdhPJwjjzv3nv7hvZLjGlUjoyYOQyxpZf9recxX0hcZis/ZbicobtUFBpp8wOCcysaZ/m8cR0ACTQ== X-Received: by 2002:a0c:fa11:: with SMTP id q17mr22585150qvn.55.1610951122782; Sun, 17 Jan 2021 22:25:22 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:22 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 3/5] build: test-driver.scm: Enable colored test results by default. Date: Mon, 18 Jan 2021 01:24:58 -0500 Message-Id: <20210118062501.27022-3-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) The Automake parallel test harness does its own smart detection of the terminal color capability and always provides the --color-tests argument to the driver. This change defaults the --color-tests argument to true when the test driver is run on its own (not via Automake). * build-aux/test-driver.scm (main): Set the default value of the --color-tests argument to true when it's not explicitly provided. --- build-aux/test-driver.scm | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 5891631da6..767a9fb25d 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -163,14 +163,17 @@ void port, which means no TRS output is logged." (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) (out (duplicate-port (current-output-port) "wl")) - (test-name (option 'test-name #f))) + (test-name (option 'test-name #f)) + (color-tests (if (assoc 'color-tests opts) + (option->boolean opts 'color-tests) + #t))) (when log (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) (redirect-port log (current-error-port))) (test-with-runner (test-runner-gnu test-name - #:color? (option->boolean opts 'color-tests) + #:color? color-tests #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) (load-from-path test-name)) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:25:34 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:34 +0000 Received: from localhost ([127.0.0.1]:46155 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyn-0003sI-MT for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:34 -0500 Received: from mail-qv1-f50.google.com ([209.85.219.50]:39113) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyk-0003rS-MY for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:31 -0500 Received: by mail-qv1-f50.google.com with SMTP id s6so7036781qvn.6 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=W1xIkCWWmhPS7flrDWDCFGiYMHzHg/vfffyAht/YNylO7dEVhwNOlJMTGPvO7nYi3I qabiVhMnYtish1xg1GIAeh/StRuuRXnw+mH4s6LajNxahvLLNi+pXeC4/TNGKuBktuel rmVhfI9HZSHaOakw6J+OuikcBjm1hwMdWy4fzTwNSWZxkDsldtZjr3iajs2txAliewRq fNvPsPOaqMLorx1v0jN6gez99ztteYxlNbEMGeE6L53vcPwo8eJwhF+wIfYLbC2JFr4W g7MUBd/OaV6pF4Bw5qhlrMUwazoVVluDV1vlpPlzqDCZXiWTuiZXJVT34tfBatBV1tJL /jKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YPxoJE0otRQnTRVl637Mr60xbpEyNm3wV1l7DNEQ+O8=; b=X0q62cFYLob9dl1ftuIBVzOB9GDppILhKZ7jj4PBdipHtmRDEt4Je4kWBvAvTcYYzG 1P3sKqEIELWzlT6+84Kf4E7VitVAfJFk7BbpthBzKrs9ZSJi/WOtDIrNkw9jrgBrwkXH 1GI8Y5q861HTj2cTVJFNlabipcmd3TFdmRKUiae7boeb36B5FyMXTshmdVW7JgTcKQxG AS/0h7rWepEzLmZnHwQbM44/xihCnrTqTXA6o2NGafjnpprGjMaz1NpuT7YTElnNQ1pT gpBEBMVDqjEbsrl2h7t+LcuLozCCK2MsEZio72wdCnETp6y5+bpIdFggI9A4frKJFCSB OCzQ== X-Gm-Message-State: AOAM530tKx4CqiT4vSfZVGIm98rqdRYrxyRA+4r8/seaWu0XUynECbbi VJuL0qW0Sfw8fOdwS811NE4Gc+bAmhg= X-Google-Smtp-Source: ABdhPJxIRNAuHwqyw1Dqqq3JmyquT6aPHkew8aQ4KPUv9lYcSpqMoHsW/QoCRaM1VOWJ7WyLVGUDCA== X-Received: by 2002:a05:6214:b2f:: with SMTP id w15mr23182347qvj.8.1610951124943; Sun, 17 Jan 2021 22:25:24 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:23 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 4/5] build: test-driver.scm: Provide the ability to filter on test case names. Date: Mon, 18 Jan 2021 01:25:00 -0500 Message-Id: <20210118062501.27022-5-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * build-aux/test-driver.scm (show-help): Add help text for the new --select and --exclude options. (%options): Add the new select and exclude options. (test-runner-gnu): Pass them to the test runner. Update doc. (test-match-name*, test-match-name*/negated, %test-match-all): New variables. (main): Compute the test specifier based on the values of the new options and apply it to the current test runner when running the test file. * doc/guix.texi (Running the Test Suite): Document the new options. --- build-aux/test-driver.scm | 66 ++++++++++++++++++++++++++++++++------- doc/guix.texi | 12 +++++++ 2 files changed, 66 insertions(+), 12 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 767a9fb25d..3ad6b0c93f 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -27,6 +27,8 @@ (use-modules (ice-9 getopt-long) (ice-9 pretty-print) + (ice-9 regex) + (srfi srfi-1) (srfi srfi-26) (srfi srfi-64)) @@ -34,14 +36,19 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] + [--select=REGEXP] [--exclude=REGEXP] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] -The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) +The '--test-name', '--log-file' and '--trs-file' options are mandatory. +The '--select' and '--exclude' options allow selecting or excluding individual +test cases via a regexp, respectively.\n")) (define %options '((test-name (value #t) (required? #t)) (log-file (value #t)) (trs-file (value #t)) + (select (value #t)) + (exclude (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -76,14 +83,22 @@ The '--test-name', '--log-file' and '--trs-file' options are mandatory.\n")) "") ;no color result))) + +;;; +;;; SRFI 64 custom test runner. +;;; + (define* (test-runner-gnu test-name #:key color? brief? (out-port (current-output-port)) - (trs-port (%make-void-port "w"))) + (trs-port (%make-void-port "w")) + select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the file name of the current the test. COLOR? specifies whether to use colors, and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged." +void port, which means no TRS output is logged. SELECT and EXCLUDE may take a +regular expression to select or exclude individual test cases based on their +names." (define (test-on-test-begin-gnu runner) ;; Procedure called at the start of an individual test case, before the @@ -148,6 +163,24 @@ void port, which means no TRS output is logged." (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) runner)) + +;;; +;;; SRFI 64 test specifiers. +;;; +(define (test-match-name* regexp) + "Return a test specifier that matches a test name against REGEXP." + (lambda (runner) + (string-match regexp (test-runner-test-name runner)))) + +(define (test-match-name*/negated regexp) + "Return a negated test specifier version of test-match-name*." + (lambda (runner) + (not (string-match regexp (test-runner-test-name runner))))) + +;;; XXX: test-match-all is a syntax, which isn't convenient to use with a list +;;; of test specifiers computed at run time. +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) + ;;; ;;; Entry point. @@ -158,15 +191,22 @@ void port, which means no TRS output is logged." (option (cut option-ref opts <> <>))) (cond ((option 'help #f) (show-help)) - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) + ((option 'version #f) (format #t "test-driver.scm ~A" 'script-version)) (else - (let ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) - (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) - (out (duplicate-port (current-output-port) "wl")) - (test-name (option 'test-name #f)) - (color-tests (if (assoc 'color-tests opts) - (option->boolean opts 'color-tests) - #t))) + (let* ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) + (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) + (out (duplicate-port (current-output-port) "wl")) + (test-name (option 'test-name #f)) + (select (option 'select #f)) + (exclude (option 'exclude #f)) + (test-specifiers (filter-map + identity + (list (and=> select test-match-name*) + (and=> exclude test-match-name*/negated)))) + (test-specifier (apply %test-match-all test-specifiers)) + (color-tests (if (assoc 'color-tests opts) + (option->boolean opts 'color-tests) + #t))) (when log (redirect-port log (current-output-port)) (redirect-port log (current-warning-port)) @@ -176,7 +216,9 @@ void port, which means no TRS output is logged." #:color? color-tests #:brief? (option->boolean opts 'brief) #:out-port out #:trs-port trs) - (load-from-path test-name)) + (test-apply test-specifier + (lambda _ + (load-from-path test-name)))) (and=> log close-port) (and=> trs close-port) (close-port out)))) diff --git a/doc/guix.texi b/doc/guix.texi index 194bb3a314..d59d2806cb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -915,6 +915,18 @@ the @code{SCM_LOG_DRIVER_FLAGS} makefile variable as in this example: make check TESTS="tests/base64.scm" SCM_LOG_DRIVER_FLAGS="--brief=no" @end example +The underlying SRFI 64 custom Automake test driver used for the 'check' +test suite (located at @file{build-aux/test-driver.scm}) also allows +selecting which test cases to run at a finer level, via its +@option{--select} and @option{--exclude} options. Here's an example, to +run all the test cases from the @file{tests/packages.scm} test file +whose names start with ``transaction-upgrade-entry'': + +@example +export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" +make check TESTS="tests/packages.scm" +@end example + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 01:25:38 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 06:25:38 +0000 Received: from localhost ([127.0.0.1]:46157 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nys-0003sZ-6f for submit@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:38 -0500 Received: from mail-qt1-f180.google.com ([209.85.160.180]:46492) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Nyl-0003rU-Qb for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 01:25:32 -0500 Received: by mail-qt1-f180.google.com with SMTP id h19so10557405qtq.13 for <45948@debbugs.gnu.org>; Sun, 17 Jan 2021 22:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dwh4qyXoSLILAiAytmhHLl4/osgRShSj7MzucfUFLug=; b=A6YjNY87nRt7JcWI3+h9+5Geh2aleRymUYlutfnzhDJE9g2bsaT355beurlkC2FBqa rft88rQcSohRVR8FOZ9cReljm1KUEZK1lXGGZEJ2raM2dFhABAwGQ74zAIgHZffC6ebN KV2+fI0ACs3lpXlxUNOWbuMLU4CRUFnAYJH9tLGgkpGEHOgeJFb79xAKvo3Z2HLoJVpP 5d5+mnniGO8ghzV5cMIopf9whQTlxVVYvyZD05FY+p8K+LKMdNcAYysB5sSZ6oBJOfEW V81IrXwvJxig7piEiRQrQ81OHKBlL3sxKAVGOUFEHBbwo3c2TV53v5sniDhoS3kbyDby r6mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dwh4qyXoSLILAiAytmhHLl4/osgRShSj7MzucfUFLug=; b=QihlMiHSlt/+mHeQ23cfXrU4UxQnDbsFED+1aIl5A+lzgiBDcgiJZwh6vP/k6beb33 DuhgnibJIyNNVJg72Gf8/NM00ibWx38mTXGOTyjbMxZvL8uu9c1ox95UcpEiBm9QVJoD GYRqldsdYnkeoQaeY/f5ZLOFmgP9z0T1MO7XkICSq9tbiqbzbZv8jjG48uhAQBcuHlPV Cs+zq9xVLLTGxz1SqVJZes9uHzspZWWiqgdlz0zcd59O5tGEy57QWxzYI5LKlH1IlGVJ tw3xlAW5wZjnvVYUrjuoptD4h8Rwmnh0EUGP7TsSlgc3Zx2l3hILPaT9XChKSq7g0ls0 jvLw== X-Gm-Message-State: AOAM530eSyRVRYZYxDRkg41xcHNmY3CbksmuXN61fFTubIBmwCze9+YS umNR7/sJ6UgyYNX6U/6Mfvk4Kf0tFVI= X-Google-Smtp-Source: ABdhPJw/ERNTEtYOqStqVWIkZZArI9GnxNkVstgu9+PydRdt0BkFet8mJIOToJ3RAohaUvmGBOSh7g== X-Received: by 2002:ac8:7153:: with SMTP id h19mr3371443qtp.118.1610951126025; Sun, 17 Jan 2021 22:25:26 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id k187sm10266726qkc.74.2021.01.17.22.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Jan 2021 22:25:25 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 5/5] build: test-driver.scm: Add a new '--errors-only' option. Date: Mon, 18 Jan 2021 01:25:01 -0500 Message-Id: <20210118062501.27022-6-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * build-aux/test-driver.scm (show-help): Add the help text for the new '--errors-only' option. (%options): Add the errors-only option. (test-runner-gnu): Add the errors-only? parameter and update doc. Move the logging of the test data after the test has completed, so a choice can be made whether to keep it or discard it based on the value of the test result. (main): Pass the errors-only? option to the driver. * doc/guix.texi (Running the Test Suite): Document the new option. --- build-aux/test-driver.scm | 73 ++++++++++++++++++++++----------------- doc/guix.texi | 12 +++++++ 2 files changed, 54 insertions(+), 31 deletions(-) diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm index 3ad6b0c93f..3f593abc7b 100644 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -36,12 +36,15 @@ (display "Usage: test-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] - [--select=REGEXP] [--exclude=REGEXP] + [--select=REGEXP] [--exclude=REGEXP] [--errors-only={yes|no}] [--enable-hard-errors={yes|no}] [--brief={yes|no}}] [--] TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS] The '--test-name', '--log-file' and '--trs-file' options are mandatory. The '--select' and '--exclude' options allow selecting or excluding individual -test cases via a regexp, respectively.\n")) +test cases via a regexp, respectively. The '--errors-only' option can be set +to \"yes\" to limit the logged test case metadata to only those test cases +that failed. When set to \"yes\", the '--brief' option disables printing the +individual test case result to the console.\n")) (define %options '((test-name (value #t) (required? #t)) @@ -49,6 +52,7 @@ test cases via a regexp, respectively.\n")) (trs-file (value #t)) (select (value #t)) (exclude (value #t)) + (errors-only (value #t)) (color-tests (value #t)) (expect-failure (value #t)) ;XXX: not implemented yet (enable-hard-errors (value #t)) ;not implemented in SRFI-64 @@ -88,27 +92,26 @@ test cases via a regexp, respectively.\n")) ;;; SRFI 64 custom test runner. ;;; -(define* (test-runner-gnu test-name #:key color? brief? +(define* (test-runner-gnu test-name #:key color? brief? errors-only? (out-port (current-output-port)) (trs-port (%make-void-port "w")) select exclude) "Return an custom SRFI-64 test runner. TEST-NAME is a string specifying the -file name of the current the test. COLOR? specifies whether to use colors, -and BRIEF?, well, you know. OUT-PORT and TRS-PORT must be output ports. -OUT-PORT defaults to the current output port, while TRS-PORT defaults to a -void port, which means no TRS output is logged. SELECT and EXCLUDE may take a -regular expression to select or exclude individual test cases based on their -names." - - (define (test-on-test-begin-gnu runner) - ;; Procedure called at the start of an individual test case, before the - ;; test expression (and expected value) are evaluated. - (let ((result (cute assq-ref (test-result-alist runner) <>))) - (format #t "test-name: ~A~%" (result 'test-name)) - (format #t "location: ~A~%" - (string-append (result 'source-file) ":" - (number->string (result 'source-line)))) - (test-display "source" (result 'source-form) #:pretty? #t))) +file name of the current the test. COLOR? specifies whether to use colors. +When BRIEF? is true, the individual test cases results are masked and only the +summary is shown. ERRORS-ONLY? reduces the amount of test case metadata +logged to only that of the failed test cases. OUT-PORT and TRS-PORT must be +output ports. OUT-PORT defaults to the current output port, while TRS-PORT +defaults to a void port, which means no TRS output is logged. SELECT and +EXCLUDE may take a regular expression to select or exclude individual test +cases based on their names." + + (define (test-skipped? runner) + (eq? 'skip (test-result-kind runner))) + + (define (test-failed? runner) + (not (or (test-passed? runner) + (test-skipped? runner)))) (define (test-on-test-end-gnu runner) ;; Procedure called at the end of an individual test case, when the result @@ -116,21 +119,29 @@ names." (let* ((results (test-result-alist runner)) (result? (cut assq <> results)) (result (cut assq-ref results <>))) - (unless brief? + (unless (or brief? (and errors-only? (test-skipped? runner))) ;; Display the result of each test case on the console. (format out-port "~A: ~A - ~A~%" (result->string (test-result-kind runner) #:colorize? color?) test-name (test-runner-test-name runner))) - (when (result? 'expected-value) - (test-display "expected-value" (result 'expected-value))) - (when (result? 'expected-error) - (test-display "expected-error" (result 'expected-error) #:pretty? #t)) - (when (result? 'actual-value) - (test-display "actual-value" (result 'actual-value))) - (when (result? 'actual-error) - (test-display "actual-error" (result 'actual-error) #:pretty? #t)) - (format #t "result: ~a~%" (result->string (result 'result-kind))) - (newline) + + (unless (and errors-only? (not (test-failed? runner))) + (format #t "test-name: ~A~%" (result 'test-name)) + (format #t "location: ~A~%" + (string-append (result 'source-file) ":" + (number->string (result 'source-line)))) + (test-display "source" (result 'source-form) #:pretty? #t) + (when (result? 'expected-value) + (test-display "expected-value" (result 'expected-value))) + (when (result? 'expected-error) + (test-display "expected-error" (result 'expected-error) #:pretty? #t)) + (when (result? 'actual-value) + (test-display "actual-value" (result 'actual-value))) + (when (result? 'actual-error) + (test-display "actual-error" (result 'actual-error) #:pretty? #t)) + (format #t "result: ~a~%" (result->string (result 'result-kind))) + (newline)) + (format trs-port ":test-result: ~A ~A~%" (result->string (test-result-kind runner)) (test-runner-test-name runner)))) @@ -157,7 +168,6 @@ names." #f)) (let ((runner (test-runner-null))) - (test-runner-on-test-begin! runner test-on-test-begin-gnu) (test-runner-on-test-end! runner test-on-test-end-gnu) (test-runner-on-group-end! runner test-on-group-end-gnu) (test-runner-on-bad-end-name! runner test-on-bad-end-name-simple) @@ -215,6 +225,7 @@ names." (test-runner-gnu test-name #:color? color-tests #:brief? (option->boolean opts 'brief) + #:errors-only? (option->boolean opts 'errors-only) #:out-port out #:trs-port trs) (test-apply test-specifier (lambda _ diff --git a/doc/guix.texi b/doc/guix.texi index d59d2806cb..ac20a541f5 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -927,6 +927,18 @@ export SCM_LOG_DRIVER_FLAGS="--select=^transaction-upgrade-entry" make check TESTS="tests/packages.scm" @end example +Those wishing to inspect the results of failed tests directly from the +command line can add the @option{--errors-only=yes} option to the +@code{SCM_LOG_DRIVER_FLAGS} makefile variable and set the @code{VERBOSE} +Automake makefile variable, as in: + +@example +make check SCM_LOG_DRIVER_FLAGS="--brief=no --errors-only=yes" VERBOSE=1 +@end example + +@xref{Parallel Test Harness,,,automake,GNU Automake} for more +information about the Automake Parallel Test Harness. + Upon failure, please email @email{bug-guix@@gnu.org} and attach the @file{test-suite.log} file. Please specify the Guix version being used as well as version numbers of the dependencies (@pxref{Requirements}) in -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 09:37:16 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 14:37:16 +0000 Received: from localhost ([127.0.0.1]:46797 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Vee-0005TD-Ig for submit@debbugs.gnu.org; Mon, 18 Jan 2021 09:37:16 -0500 Received: from mail-qv1-f51.google.com ([209.85.219.51]:34532) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Vec-0005T0-W7 for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 09:37:15 -0500 Received: by mail-qv1-f51.google.com with SMTP id h13so7575089qvo.1 for <45948@debbugs.gnu.org>; Mon, 18 Jan 2021 06:37:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:references:date:in-reply-to:message-id:user-agent :mime-version; bh=1DhYug7f8oI7ZefV7lVbAqMI87c6HCcQ+YPGHuJ4t3g=; b=hTVIqWxxYE7sRkIR80EuBH2X72oqWQfqXZcLGnl7RtIgnDLCfdlsK32wEiiZoQnski Nq4EvgX4LpvxljvGSMI3G3mz7Dm6Tb7WdEhmT2OKa0RDq6dZ1b9E9JW7bXdfBc4RYB+8 hupFEkCazeNC6pZ7U8DhOY/NY91lLoRjEJmT1h5zKK+RSwaToTUJpwQg1tzSSH/PuHo1 RVd4EXNm2yr6Iyyt/fuAQNIsrDnXlr9JL5Q0zlmVVdicvPZs+DyG7RoYG7Ce/FNPYtQe o17St06GU0/vlJHnb2gmNjIpmjD0qEZ4vOCX0kbre/N3ngAHXL6W7s+8Fjat6WBEhqux D3fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=1DhYug7f8oI7ZefV7lVbAqMI87c6HCcQ+YPGHuJ4t3g=; b=LuAH8ApmsBlWkatCWIL+XLiM6PDVdOCiK5l6ar2NwXImoFkt9FOhdtHmVcKyCX5LzE emstTHd6B8eb2Ol4fNex+nwSqHFhFZB4Hr+JNOT4j53CJk4RB2iK+7Mwd3AQGemsU+8t BVtZ3mpXeSmTl7cEE4komm1dk0n1r/wk7ZwiJtRQCx4+EStiZLhlFelmbPhx4azQesxV wRdMkpbD6l5YxJ26P/zQqe4pxMY8nAjPGz+V5I0o7ufeWADV+J7ogHL6kwW2lAk1xGJu irOCVimzqQDVaLjGkwiJE5BtqYq+xMxoJ+oIaPdzG755/fRHQ+gN9b+bSNAmpt6QXQZn A62A== X-Gm-Message-State: AOAM533uEtQ+F952eTnrhYOPx50Jdo1nJoAcoIMxgyTD4oz7VQlkylyF CMvN4Eu+AzVLxSgihV0QBhjNUxPBzfg= X-Google-Smtp-Source: ABdhPJwfi1NuPdB0YIepEL1ruatrT5Kjd7LpM3MwdDKtGqiACnHUCt1uJJRfXOblKvZMUV7hy1xuZQ== X-Received: by 2002:a0c:e90a:: with SMTP id a10mr24712395qvo.38.1610980629249; Mon, 18 Jan 2021 06:37:09 -0800 (PST) Received: from hurd (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id w8sm10412785qts.50.2021.01.18.06.37.08 for <45948@debbugs.gnu.org> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jan 2021 06:37:08 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-2-maxim.cournoyer@gmail.com> Date: Mon, 18 Jan 2021 09:37:07 -0500 In-Reply-To: <20210118062501.27022-2-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Mon, 18 Jan 2021 01:24:57 -0500") Message-ID: <87r1mijoq4.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello, Maxim Cournoyer writes: > This is clearer than failing to open an empty string file name, when the > option is not provided. > > * build-aux/test-driver.scm (%options)[test-name]: Set the required? property > to #t. I just noticed this doesn't play well with the --help or --version options. So this patch should be dropped. Thanks, Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 09:38:47 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 14:38:47 +0000 Received: from localhost ([127.0.0.1]:46801 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Vg6-0005VH-UJ for submit@debbugs.gnu.org; Mon, 18 Jan 2021 09:38:47 -0500 Received: from mail-qk1-f170.google.com ([209.85.222.170]:40161) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Vg5-0005V4-G1 for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 09:38:45 -0500 Received: by mail-qk1-f170.google.com with SMTP id z11so18795868qkj.7 for <45948@debbugs.gnu.org>; Mon, 18 Jan 2021 06:38:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:references:date:in-reply-to:message-id:user-agent :mime-version; bh=rkxJQufFbrA87NKaAIwnBfyfWxzdKP6hnvQ0CgAR/W0=; b=fZ9QsYmTQpgznAPie0+9C9Y4HdCtvsvXOZmB/a72ZndRRwBo152hyIKXRVEdExngBR tdRBWfxSNv9AH88krchgkVP5vGx4UN/GBG4O0lzExZXM4MHQO32aJ4rtDk9pLPoP3QKV aNCTcYH/cJGR0mqW1IwT9SZtL41hVj71pgf+L3zA+Y6RrsARenayMZ76+gJgcktmAyfU rSiot+X7bikZuglmGP/TpcRYyhddHZaFcJ4Qlin40Xhv+/6nEwNlZQlU89YTb0uEH5PQ PyAm+oIwE3lxeQH5zthXQqNEI47z6e3+sa1ctHJhB/0MfOLaRS4GG+FzHfjuafKiocSE J25w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=rkxJQufFbrA87NKaAIwnBfyfWxzdKP6hnvQ0CgAR/W0=; b=gj9L4/ri82rJBQKdbCIpULTd5ij1mXXs78NApHALCiQPcHnkb3/YNWhxb7ag0HOcf8 GxDuqnvdA/RXc/pf5f2Jzx6rnnYwH+iX31mhySqPdfL5Zz6icNNXueZsRE3oa+G9R17a c+I5wjptY/kEs4pCgsEVKblutnaw24SfpP61Bo7fW2/2WxlFB0jApUKvdtdX/yg6lxug MJLE9sX30QnzBik9PCjI3t6HeuijE/MAJBxepIPqaHi0QIb83hM2Cl6CnocujbTx+ZE6 1xH1OXUZe3PUnFC4vBSh/LpQPYVeNoj36c9YEPsyviDwynQThgiypGvD3gsAmwblp1Ms SUxA== X-Gm-Message-State: AOAM533BuXcuFZ7pmh+8NweQFxVxBvL66z0kjYJhCqQzbP1gI1lEUGB1 IrYoMY0lBsXkSHywGAyy0PKBRsOGxNs= X-Google-Smtp-Source: ABdhPJwDd5rYCE1sfdw93TOr/BxY30kciiNugqIRwoV4nIRze9PTwMEGbmEkATfKLdVCly+Me2IgKw== X-Received: by 2002:ae9:e308:: with SMTP id v8mr24488048qkf.339.1610980719854; Mon, 18 Jan 2021 06:38:39 -0800 (PST) Received: from hurd (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id e12sm10758244qka.82.2021.01.18.06.38.39 for <45948@debbugs.gnu.org> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jan 2021 06:38:39 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-4-maxim.cournoyer@gmail.com> Date: Mon, 18 Jan 2021 09:38:38 -0500 In-Reply-To: <20210118062501.27022-4-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Mon, 18 Jan 2021 01:24:59 -0500") Message-ID: <87mtx6jonl.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello, Maxim Cournoyer writes: [...] > ((option 'help #f) (show-help)) > - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) > + ((option 'version #f) (format #t "test-driver.scm ~A" 'script-version)) This part of the diff is erroneous. Please disregard. Thanks, Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 09:42:22 2021 Received: (at 45948) by debbugs.gnu.org; 18 Jan 2021 14:42:22 +0000 Received: from localhost ([127.0.0.1]:46815 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Vja-0005bI-KS for submit@debbugs.gnu.org; Mon, 18 Jan 2021 09:42:22 -0500 Received: from mail-qt1-f175.google.com ([209.85.160.175]:44437) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1VjZ-0005b6-NS for 45948@debbugs.gnu.org; Mon, 18 Jan 2021 09:42:21 -0500 Received: by mail-qt1-f175.google.com with SMTP id r9so11357764qtp.11 for <45948@debbugs.gnu.org>; Mon, 18 Jan 2021 06:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=x25XXMmYl+uQGNGn4N/YOegz++Oij/4uayVl8eVhpWk=; b=pa+GDnBS1Jsa2HSZmFMaKbXwrgces8Fk9Gm3RssK5Wj4YMzXEgKUlCAyU9jnX+gfzr gGzd+grzP2Ee+LmQQfMXL67C2Yc5fzH+BzFFNl5yyG0UKqzAqO3uYzHMEEv95x7ymQS2 lCcsFtjVV8qDdnjEUHpF/z3kewnvthNecp7uka+Z+RXNpv++FhMcvTqjMnRd8UgmSB/E x4tGnt8jclCc8FZ5PFtzwhpESlvy61GiGxemq2/tvDM1No7WDzTnnYvakUylG0I6ZKGh ydYS6xi1Qo3gYOoiP6gSvIiTu+1S9O6ythhlbIcC96FF2uZpMcWj9xtGMahWaJzGg1Ef zH3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x25XXMmYl+uQGNGn4N/YOegz++Oij/4uayVl8eVhpWk=; b=VID4lEQGJ3rWJzqdJDbzfbq7glr6Cg08OdtLhiTFT9t8ID1ZZZZoQUvofuy7xryVha knpYt9Ls7XI8ssQJV8hUMKd8z7ZAvJ4WWZn73DnLc3T7Euz2FtV9+eBpterFRhs3Izzy ecE2lrBtgJL3wotygZAdWaQPdEe+S7r2A6hfYjpVnsMto8R9TvUoFTF8SfIqZoJXnhWF OIJwIHNWpj6CcFffgB4yOm9tmrs7ilWkFtrY8UfenNc/KmbXBOzJNmrzgJXE40pf+0di 18VOom4Tdl1xXm3UQtQlh08gb1S9DoR89AoQ8LyQQqgyAqbhf71EjxnAr8tmYUYYLqPM z9qQ== X-Gm-Message-State: AOAM532ZYo0foAxi+ibpgm6qwr+Wn5Ozco5umuuj/RxVO9f4iJpbxnMu DRPCSooiRd0FWIUnjvkjUex4gPM6ESI= X-Google-Smtp-Source: ABdhPJy3sM8CDODZOp76/FWfFyrq03qOVgUrTJfXJn6meqp4+xpRml/XT/18OxVOsd701M6OpgVlhw== X-Received: by 2002:ac8:4f51:: with SMTP id i17mr7935726qtw.78.1610980936098; Mon, 18 Jan 2021 06:42:16 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id n5sm10888309qkh.126.2021.01.18.06.42.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jan 2021 06:42:15 -0800 (PST) From: Maxim Cournoyer To: 45948@debbugs.gnu.org Subject: [PATCH 6/5] build: test-driver.scm: Allow running as a standalone script. Date: Mon, 18 Jan 2021 09:42:01 -0500 Message-Id: <20210118144201.11923-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * build-aux/test-driver.scm: Add an exec-based shebang and set the script executable bit. (main): Insert a newline after the version string is printed with --version. --- build-aux/test-driver.scm | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) mode change 100644 => 100755 build-aux/test-driver.scm diff --git a/build-aux/test-driver.scm b/build-aux/test-driver.scm old mode 100644 new mode 100755 index aa34e669ff..2d0649ef37 --- a/build-aux/test-driver.scm +++ b/build-aux/test-driver.scm @@ -1,6 +1,9 @@ +#!/bin/sh +exec guile --no-auto-compile -e main -s "$0" "$@" +!# ;;;; test-driver.scm - Guile test driver for Automake testsuite harness -(define script-version "2021-01-18.06") ;UTC +(define script-version "2021-01-18.14") ;UTC ;;; Copyright © 2015, 2016 Mathieu Lirzin ;;; Copyright © 2021 Maxim Cournoyer @@ -201,7 +204,7 @@ cases based on their names." (option (cut option-ref opts <> <>))) (cond ((option 'help #f) (show-help)) - ((option 'version #f) (format #t "test-driver.scm ~A" script-version)) + ((option 'version #f) (format #t "test-driver.scm ~A~%" script-version)) (else (let* ((log (and=> (option 'log-file #f) (cut open-file <> "w0"))) (trs (and=> (option 'trs-file #f) (cut open-file <> "wl"))) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 30 16:32:48 2021 Received: (at 45948) by debbugs.gnu.org; 30 Jan 2021 21:32:48 +0000 Received: from localhost ([127.0.0.1]:55921 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5xrK-0007Yu-1f for submit@debbugs.gnu.org; Sat, 30 Jan 2021 16:32:48 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53096) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5xrI-0007Yi-Ng for 45948@debbugs.gnu.org; Sat, 30 Jan 2021 16:32:44 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51658) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5xrD-0005x6-H9; Sat, 30 Jan 2021 16:32:39 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48876 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l5xrC-0000eM-Mr; Sat, 30 Jan 2021 16:32:39 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118061853.26808-1-maxim.cournoyer@gmail.com> <20210118062501.27022-1-maxim.cournoyer@gmail.com> Date: Sat, 30 Jan 2021 22:32:37 +0100 In-Reply-To: <20210118062501.27022-1-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Mon, 18 Jan 2021 01:24:56 -0500") Message-ID: <87k0rup0ui.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 45948 Cc: 45948@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi! Maxim Cournoyer skribis: > This makes it easier (and less surprising) for users to experiment with t= he > custom Scheme test driver directly. The behavior is unchanged from Autom= ake's > point of view. > > * build-aux/test-driver.scm (main): Make the --log-file and --trs-file > arguments optional. Only open, redirect and close a port to a log file w= hen > the --log-file option is provided. Only open and close a port to a trs f= ile > when the --trs-file option is provided. > (test-runner-gnu): Set OUT-PORT parameter default value to the current ou= tput > port. Set the TRS-PORT parameter default value to a void port. Update d= oc. Not tested, but LGTM! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 30 16:34:59 2021 Received: (at 45948) by debbugs.gnu.org; 30 Jan 2021 21:34:59 +0000 Received: from localhost ([127.0.0.1]:55925 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5xtS-0007c8-ND for submit@debbugs.gnu.org; Sat, 30 Jan 2021 16:34:58 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53320) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5xtQ-0007bt-Ut for 45948@debbugs.gnu.org; Sat, 30 Jan 2021 16:34:57 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51673) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5xtL-0006su-E3; Sat, 30 Jan 2021 16:34:51 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48878 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l5xtL-0000I8-2N; Sat, 30 Jan 2021 16:34:51 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> Date: Sat, 30 Jan 2021 22:34:49 +0100 In-Reply-To: <20210118062501.27022-5-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Mon, 18 Jan 2021 01:25:00 -0500") Message-ID: <87eei2p0qu.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 45948 Cc: 45948@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi! Maxim Cournoyer skribis: > * build-aux/test-driver.scm (show-help): Add help text for the new --sele= ct > and --exclude options. > (%options): Add the new select and exclude options. > (test-runner-gnu): Pass them to the test runner. Update doc. > (test-match-name*, test-match-name*/negated, %test-match-all): New variab= les. > (main): Compute the test specifier based on the values of the new options= and > apply it to the current test runner when running the test file. > * doc/guix.texi (Running the Test Suite): Document the new options. I never felt the need for this since most individual files run quickly enough (and those that don=E2=80=99t should be optimized=E2=80=A6), but it = can be useful. > +;;; XXX: test-match-all is a syntax, which isn't convenient to use with = a list > +;;; of test specifiers computed at run time. > +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) Since this is an internal procedure in Guile that could vanish anytime, I recommend copying it here (it=E2=80=99s just 9 lines). > +The underlying SRFI 64 custom Automake test driver used for the 'check' > +test suite (located at @file{build-aux/test-driver.scm}) also allows Maybe shorten to =E2=80=9CThe underlying test driver (located at @file{build-aux/test-driver.scm}) also allows=E2=80=9D. I didn=E2=80=99t test it, but it otherwise LGTM. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 30 16:39:26 2021 Received: (at 45948) by debbugs.gnu.org; 30 Jan 2021 21:39:26 +0000 Received: from localhost ([127.0.0.1]:55929 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5xxm-0007ic-AJ for submit@debbugs.gnu.org; Sat, 30 Jan 2021 16:39:26 -0500 Received: from eggs.gnu.org ([209.51.188.92]:53776) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l5xxk-0007iP-EQ for 45948@debbugs.gnu.org; Sat, 30 Jan 2021 16:39:25 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51695) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5xxf-0000Re-0k; Sat, 30 Jan 2021 16:39:19 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=48880 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l5xxe-0000al-Gc; Sat, 30 Jan 2021 16:39:18 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-3-maxim.cournoyer@gmail.com> Date: Sat, 30 Jan 2021 22:39:17 +0100 In-Reply-To: <20210118062501.27022-3-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Mon, 18 Jan 2021 01:24:58 -0500") Message-ID: <875z3ep0je.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 45948 Cc: 45948@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Maxim Cournoyer skribis: > The Automake parallel test harness does its own smart detection of the > terminal color capability and always provides the --color-tests argument = to > the driver. This change defaults the --color-tests argument to true when= the > test driver is run on its own (not via Automake). > > * build-aux/test-driver.scm (main): Set the default value of the --color-= tests > argument to true when it's not explicitly provided. It=E2=80=99s a small change to why not, but=E2=80=A6 =E2=80=A6 the test driver is not meant to be used on its own though, it=E2= =80=99s really a test driver for Automake. Plus, in most projects, part of the test environment is defined in =E2=80=98Makefile.am=E2=80=99; I wouldn=E2=80=99t want to process bug repor= ts because people thought they could try and run tests without =E2=80=9Cmake check=E2=80=9D. Thoughts? As a side note, this test driver is bundled in quite a few Guile packages. Longer-term, it would be nice to consider maintaining it as part of Automake. Now is maybe a good time since there=E2=80=99s on-going maintenance work happening in Autoconf (and soon Automake and Libtool) after a long hiatus. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 31 21:47:57 2021 Received: (at 45948) by debbugs.gnu.org; 1 Feb 2021 02:47:57 +0000 Received: from localhost ([127.0.0.1]:58416 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6PFt-0007xi-Kc for submit@debbugs.gnu.org; Sun, 31 Jan 2021 21:47:57 -0500 Received: from mail-qk1-f178.google.com ([209.85.222.178]:40958) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6PFq-0007xT-1h for 45948@debbugs.gnu.org; Sun, 31 Jan 2021 21:47:55 -0500 Received: by mail-qk1-f178.google.com with SMTP id u20so14941751qku.7 for <45948@debbugs.gnu.org>; Sun, 31 Jan 2021 18:47:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=lloc90Tb9ZLgN7X6/o/nf/7KwpKYN8IG+CUFs1kDYqA=; b=GJz7PcPmx9lv/R8VSclGZxUNRnEcVWAxHq93Mpdj0ZuGYQDLDEDfyA030B6qI8sJ1S qOqbl9w8riZkr97fF5oWiimWfGGHlHqTxNScpPQ1ioVqMueqP5TssqWWujXZWXnu4pSt f/J04FkC7r2ApgkZ5/Lup2mAclnwhPVcNsYrCcUSzd1DMsSuNN/HQZryWmMk06VCUsBF /K68Tmp6oj1DFPJ+JmvNTbjkwA/FuxC9EGjgW6TJsYuye4p5+IIbtCmWPPALtv3G2p0P lSdbtljHggCk2UeC8njhbogQZB6P4FXUEcE1O5FvXBFp6trs9Hoo4BA2pcQxd/7mxbir xwKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=lloc90Tb9ZLgN7X6/o/nf/7KwpKYN8IG+CUFs1kDYqA=; b=n/UO3okqyJPjFWcl+KlnukuyzmNMY/LgiWAEDHH+V42AHB1uP8EMANcSuFVMCxmmj5 t2PiL1AYvd4lhZazWN0o4fFDK33ZGfP61vCYnaF9RqSgmqAWrVtNBPZPpDu4d7qz1HVs 5hb4qrw4o6lbwm24ifrI6tAP2GF4z1DR6ZSsVeeM6UkPUs3PuuyJJ4A+KxX3DWgShLVh IaTEX4K2l5MS2VRSTg06mXqXH/TLtcLYxpNzEBRnwiH6OHz4xGElPKZp7evXB4oonOP2 MRwzLXhDI4F/Ph+h/eXvC5pJuFz8QkbX3Hq8BklXXP9+MLG/2Vjm5M4mPAiWoKHnlzCN 684Q== X-Gm-Message-State: AOAM530PV0K7xiSyW0WSQhHBF5Qla9AkvAvf/Zxk1mEu027DdSWeEjIR uT+Nnn2e9o5FOPrMtD6s3m6Qyt5/5BE= X-Google-Smtp-Source: ABdhPJwBF/wEPHwPuMiJOD30hWlRLTqkx3fnbERR3T89clCG7Rq0Eak+vu2Egd1CCCmGl359wKPbjg== X-Received: by 2002:a37:591:: with SMTP id 139mr14509151qkf.493.1612147668038; Sun, 31 Jan 2021 18:47:48 -0800 (PST) Received: from hurd (dsl-205-236-230-168.b2b2c.ca. [205.236.230.168]) by smtp.gmail.com with ESMTPSA id i17sm13697514qtg.77.2021.01.31.18.47.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jan 2021 18:47:47 -0800 (PST) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-3-maxim.cournoyer@gmail.com> <875z3ep0je.fsf_-_@gnu.org> Date: Sun, 31 Jan 2021 21:47:46 -0500 In-Reply-To: <875z3ep0je.fsf_-_@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s?= =?utf-8?Q?=22's?= message of "Sat, 30 Jan 2021 22:39:17 +0100") Message-ID: <87r1m0sdv1.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948 Cc: 45948@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, Ludovic Court=C3=A8s writes: > Maxim Cournoyer skribis: > >> The Automake parallel test harness does its own smart detection of the >> terminal color capability and always provides the --color-tests argument= to >> the driver. This change defaults the --color-tests argument to true whe= n the >> test driver is run on its own (not via Automake). >> >> * build-aux/test-driver.scm (main): Set the default value of the --color= -tests >> argument to true when it's not explicitly provided. > > It=E2=80=99s a small change to why not, but=E2=80=A6 > > =E2=80=A6 the test driver is not meant to be used on its own though, it= =E2=80=99s really > a test driver for Automake. The interface via make is not terribly convenient/discoverable; I find having the test-driver produce useful output on its own handy for hacking on the thing hands on. > Plus, in most projects, part of the test environment is defined in > =E2=80=98Makefile.am=E2=80=99; I wouldn=E2=80=99t want to process bug rep= orts because people > thought they could try and run tests without =E2=80=9Cmake check=E2=80=9D. The use of the test driver without the Automake build system is undocumented; I wouldn't be too worried about people suddenly stumbling on it. Thanks, Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 31 22:44:25 2021 Received: (at 45948-done) by debbugs.gnu.org; 1 Feb 2021 03:44:25 +0000 Received: from localhost ([127.0.0.1]:58444 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6Q8X-0000tv-7W for submit@debbugs.gnu.org; Sun, 31 Jan 2021 22:44:25 -0500 Received: from mail-qk1-f177.google.com ([209.85.222.177]:41607) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6Q8V-0000tc-8V for 45948-done@debbugs.gnu.org; Sun, 31 Jan 2021 22:44:23 -0500 Received: by mail-qk1-f177.google.com with SMTP id n15so15028780qkh.8 for <45948-done@debbugs.gnu.org>; Sun, 31 Jan 2021 19:44:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=55Bc7IBinpJTRWNoCoFxHL2ZkufaNeFGzxg5C3kE4SA=; b=ExmLePh7H2S0rfRrQl/VVcsxYn34XQNUyN4S/DAuQdsOgtWii8ZLp8+qGhPQnKDVth BDOYDRD0oKoVg+I9yRuLGCfPRtPoLQtZO2Rv0Zt42LY2rmrPpDUfatXgSQeDZ1VOdijZ /s/2gsJk8b72r1BPR1c5X848FQ3bRIjzSsuo9d+bJOzFzLkiYipi8Chk4JRsx5RJT2E6 1XnKx3GOtUxuOaF0R2d6uTi7jUhbWkXncktlygYylbDkw09mbYYSMaAZEaBVqlcKuhbI bVO3YjW0Qtk6QETYgcloen16IPzGZgLcYHo7GXSQMB0QlrafANbB9jDOdbVUmztsV0jJ aZWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=55Bc7IBinpJTRWNoCoFxHL2ZkufaNeFGzxg5C3kE4SA=; b=iO1E20tWif9Zt7/29su4ubZss5Puu2xcEgwk2Psv8m+LUcrHws3TwVc8EkhqlQtEy7 Yd90XDLEOkEyuQQdRdQYV80Ui1c4BesGR+fWeOedxZ3Z8QlO1m+tvc4Vq2W4NmRYOWrA FvpBk0Z4OCkDmA2C42YnqVi2kTJXQ5/LqbEVunICSHkSbYRlALkiw7PRaE/GFjrGwJeU vRFeLhSUW7I5i2W1pf+llsiMZkoj//6D3G7D8HdWxRmr9+2/Q9kIblk1x3Ir57kpENQ6 KdUilqkQ5nb58UQM8SG1ovVMWhJr8eSG5jgadG1oD6kqhfE4Ajb/U6bWFN3DbWzFgHn7 Vzpg== X-Gm-Message-State: AOAM532TO4vJzUs7ZriR1BTMm73RqJaElENVnUkw6XVFMF/LzhrNSgzc udU1PfgbX3laRQluYvwt8C9eGkfAvQc= X-Google-Smtp-Source: ABdhPJzpcanGF2O3/cxpaFOWASNcupLCFsQL8NnhrgXN7dfiC2Nsh6zYH4fxMuz48MB/AFFP7RiTVA== X-Received: by 2002:a37:644e:: with SMTP id y75mr14337752qkb.134.1612151057464; Sun, 31 Jan 2021 19:44:17 -0800 (PST) Received: from hurd (dsl-205-236-230-168.b2b2c.ca. [205.236.230.168]) by smtp.gmail.com with ESMTPSA id q18sm5520844qtr.74.2021.01.31.19.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jan 2021 19:44:16 -0800 (PST) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> <87eei2p0qu.fsf_-_@gnu.org> Date: Sun, 31 Jan 2021 22:44:15 -0500 In-Reply-To: <87eei2p0qu.fsf_-_@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s?= =?utf-8?Q?=22's?= message of "Sat, 30 Jan 2021 22:34:49 +0100") Message-ID: <87lfc8sb8w.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948-done Cc: 45948-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello, Ludovic Court=C3=A8s writes: [...] > I never felt the need for this since most individual files run quickly > enough (and those that don=E2=80=99t should be optimized=E2=80=A6), but i= t can be > useful. What triggered it for me was trying to iterate using tests added to the tests/packages.scm test module: --8<---------------cut here---------------start------------->8--- $ time make check TESTS=3Dtests/packages.scm VERBOSE=3D1 SCM_LOG_DRIVER_FLAGS=3D"--brief=3Dno" time make check TESTS=3Dtests/packages.scm VERBOSE=3D1 SCM_LOG_DRIVER_FLAGS= =3D"--brief=3Dno" [...] PASS: tests/packages.scm - package-patched-vulnerabilities PASS: tests/packages.scm - fold-packages PASS: tests/packages.scm - fold-packages, hidden package [... time passes ...] PASS: tests/packages.scm - fold-available-packages with/without cache [...] PASS: tests/packages.scm - find-package-locations with cache PASS: tests/packages.scm - specification->location =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D Testsuite summary for GNU Guix UNKNOWN =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D # TOTAL: 100 # PASS: 100 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D [...] real 0m46.172s user 1m4.885s sys 0m0.376s --8<---------------cut here---------------end--------------->8--- That's on the fastest machine I have access to (on my vintage desktop, it took nearly 4 minutes). The slowest test seems to be 'fold-available-packages with/without cache'. Now with the new select, one can do: --8<---------------cut here---------------start------------->8--- $ time make check TESTS=3Dtests/packages.scm VERBOSE=3D1 SCM_LOG_DRIVER_FLA= GS=3D"--brief=3Dno --select=3D'bag->derivation' --errors-only=3Dyes" [...] PASS: tests/packages.scm - bag->derivation PASS: tests/packages.scm - bag->derivation, cross-compilation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D Testsuite summary for GNU Guix UNKNOWN =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D # TOTAL: 100 # PASS: 2 # SKIP: 98 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D [...] real 0m1.569s user 0m2.382s sys 0m0.154s --8<---------------cut here---------------end--------------->8--- 1.6 s; better than 46 s! We can also check the time the suspected slow test took: $ time make check TESTS=3Dtests/packages.scm SCM_LOG_DRIVER_FLAGS=3D"--sele= ct=3D'fold-available-packages with/without cache'" [...] PASS: tests/packages.scm - fold-available-packages with/without cache =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D Testsuite summary for GNU Guix UNKNOWN =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D # TOTAL: 100 # PASS: 1 # SKIP: 99 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D [...] real 0m36.627s user 0m45.731s sys 0m0.264s So yes, this is the most expensive test of tests/packages.scm. >> +;;; XXX: test-match-all is a syntax, which isn't convenient to use with= a list >> +;;; of test specifiers computed at run time. >> +(define %test-match-all (@@ (srfi srfi-64) %test-match-all)) > > Since this is an internal procedure in Guile that could vanish anytime, > I recommend copying it here (it=E2=80=99s just 9 lines). Done! >> +The underlying SRFI 64 custom Automake test driver used for the 'check' >> +test suite (located at @file{build-aux/test-driver.scm}) also allows > > Maybe shorten to =E2=80=9CThe underlying test driver (located at > @file{build-aux/test-driver.scm}) also allows=E2=80=9D. I see value in explicitly stating what it is, as it took me some effort to be able to answer that question when I started looking at it (the test driver). I've now pushed this series to master; thank you for the review! Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Feb 01 17:18:39 2021 Received: (at 45948-done) by debbugs.gnu.org; 1 Feb 2021 22:18:39 +0000 Received: from localhost ([127.0.0.1]:33460 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6hWn-0004rG-Rh for submit@debbugs.gnu.org; Mon, 01 Feb 2021 17:18:39 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34674) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6hWk-0004r0-H3 for 45948-done@debbugs.gnu.org; Mon, 01 Feb 2021 17:18:37 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34520) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l6hWf-0002DA-21; Mon, 01 Feb 2021 17:18:29 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=52238 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l6hWe-0007nM-LR; Mon, 01 Feb 2021 17:18:28 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> <87eei2p0qu.fsf_-_@gnu.org> <87lfc8sb8w.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 13 =?utf-8?Q?Pluvi=C3=B4se?= an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 01 Feb 2021 23:18:27 +0100 Message-ID: <87czxjh1os.fsf@inria.fr> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 45948-done Cc: 45948-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hello! Maxim Cournoyer skribis: > Ludovic Court=C3=A8s writes: > > [...] > >> I never felt the need for this since most individual files run quickly >> enough (and those that don=E2=80=99t should be optimized=E2=80=A6), but = it can be >> useful. > > What triggered it for me was trying to iterate using tests added to the > tests/packages.scm test module: [...] > 1.6 s; better than 46 s! Quite a difference, indeed! Some tests, like =E2=80=98tests/store.scm=E2=80=99, invoke the GC (via =E2= =80=98delete-paths=E2=80=99 calls in particular), and this is a bad idea because it takes ages. What I meant by =E2=80=9Cshould be optimized=E2=80=9D is that these tests s= hould be tweaked to avoid invoking the GC as much as possible. That=E2=80=99s not the case of =E2=80=98tests/packages.scm=E2=80=99 though.= Which gives me an idea: what would it take to modify the test driver so it can print the time spent in each test? :-) That would probably prove helpful to optimize core Guix. In many cases, I also run the one test I=E2=80=99m interested in through Ge= iser, which gives an optimally fast feedback loop. > We can also check the time the suspected slow test took: > > $ time make check TESTS=3Dtests/packages.scm SCM_LOG_DRIVER_FLAGS=3D"--se= lect=3D'fold-available-packages with/without cache'" > [...] > PASS: tests/packages.scm - fold-available-packages with/without cache Ah ha! Turns out a large part of the time was due to the O(n=C2=B2) behavi= or of the various list operations (the list of packages is big enough!). Fixed in 73744725dd0a65cddaa9251f104f17ca27756479. >>> +The underlying SRFI 64 custom Automake test driver used for the 'check' >>> +test suite (located at @file{build-aux/test-driver.scm}) also allows >> >> Maybe shorten to =E2=80=9CThe underlying test driver (located at >> @file{build-aux/test-driver.scm}) also allows=E2=80=9D. > > I see value in explicitly stating what it is, as it took me some effort > to be able to answer that question when I started looking at it (the > test driver). Agreed. It just seemed to me that we were mentioning three new concepts/tools in passing: SRFI-64, Automake, and test drivers. > I've now pushed this series to master; thank you for the review! Thank you! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 02 00:47:22 2021 Received: (at 45948-done) by debbugs.gnu.org; 2 Feb 2021 05:47:22 +0000 Received: from localhost ([127.0.0.1]:33875 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6oX3-00011n-VA for submit@debbugs.gnu.org; Tue, 02 Feb 2021 00:47:22 -0500 Received: from mail-qk1-f176.google.com ([209.85.222.176]:43856) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6oX0-00011a-VV for 45948-done@debbugs.gnu.org; Tue, 02 Feb 2021 00:47:21 -0500 Received: by mail-qk1-f176.google.com with SMTP id a12so18784266qkh.10 for <45948-done@debbugs.gnu.org>; Mon, 01 Feb 2021 21:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=7KtunbuFuJ2Pl5Wpy9MUW5C/wLK/+j6BuKaLxGUHvDg=; b=RhM4BXL3joFN4R0PyN23e4AQMAsAci39W4k/meOXrS2+rwjK8LIbQ5GEdEVlnu1T6m ba+L/NlNFDIjLGFh2nyhDSBVJODd0F5adKGw7JSOxCIfoEjApFmkfyOTdszThXhnp0s+ i3P61/r8HKX39W1EwgXyqo9WT0xq5lsUtdGkxnQjHuWs5G50YG3lWl+Wgjj1wBV07BQF fEgokneZjiS91we2qYUrJ+DHWCq2C2aRwCo43UcpfH6ZqlP3I/+AqjS1tkuR3sc06DqY OT7GXlYCMDPQlQjpFmbYsmfEspJPz6l16gMrQqrrhi8mhAy9y0zl33PBjP5uYyvTC1Lr BiiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=7KtunbuFuJ2Pl5Wpy9MUW5C/wLK/+j6BuKaLxGUHvDg=; b=BGfiEcliY2nPEMVpkJ9pFBNtZjez0Yn+KglPNtG5DasaZYV+KPCdxUNHfdGnaO60yN mOGQgwOAYKVVVD6TFyaMwhysX2jNjCmbZ9M2e61s4GMfdA+4qEs452j7BPuz8qnRvKUS OB5TVp5Iwn0r2xAX3NPS8Yl20wzdh6JrZdYJ+e8GtUmn3sADb3/s9CorCZzU/1krFo84 MsQLbm9zn2PZ0VkSrjuf1dkCZu0DlRkxPHDUs0bPyJ1dX+t6UAf4c7jGpW1ngmLwnnQX Qmh3wgR7iMgIPyp0R5xcw5xlhIRa7W+Dxm75/wQp2n3Acq0bAYkeVAwMIGXfn4dcWLZD 4lPw== X-Gm-Message-State: AOAM532fO4/FKvZo1XikodR8A4b7LbIsPEDzmGXwMH1JmqpYIifiLzfJ VcRaLSpxv6P/1VijmV/AU/AzATsVz8Fj1Q== X-Google-Smtp-Source: ABdhPJx8PYsJ1fpZbhkpUHiWoNB/GiSN/K9lQYJX54bPFOjxR1h+9QOaEQY4eubdqVjDVOhcs+8qog== X-Received: by 2002:a37:6609:: with SMTP id a9mr19340012qkc.123.1612244833067; Mon, 01 Feb 2021 21:47:13 -0800 (PST) Received: from hurd (dsl-205-236-230-168.b2b2c.ca. [205.236.230.168]) by smtp.gmail.com with ESMTPSA id u133sm11381138qka.116.2021.02.01.21.47.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 21:47:12 -0800 (PST) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45948: [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> <87eei2p0qu.fsf_-_@gnu.org> <87lfc8sb8w.fsf@gmail.com> <87czxjh1os.fsf@inria.fr> Date: Tue, 02 Feb 2021 00:47:11 -0500 In-Reply-To: <87czxjh1os.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Mon, 01 Feb 2021 23:18:27 +0100") Message-ID: <87h7mvqaw0.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948-done Cc: 45948-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, Ludovic Court=C3=A8s writes: > Hello! > > Maxim Cournoyer skribis: > >> Ludovic Court=C3=A8s writes: >> >> [...] >> >>> I never felt the need for this since most individual files run quickly >>> enough (and those that don=E2=80=99t should be optimized=E2=80=A6), but= it can be >>> useful. >> >> What triggered it for me was trying to iterate using tests added to the >> tests/packages.scm test module: > > [...] > >> 1.6 s; better than 46 s! > > Quite a difference, indeed! > > Some tests, like =E2=80=98tests/store.scm=E2=80=99, invoke the GC (via = =E2=80=98delete-paths=E2=80=99 > calls in particular), and this is a bad idea because it takes ages. > What I meant by =E2=80=9Cshould be optimized=E2=80=9D is that these tests= should be > tweaked to avoid invoking the GC as much as possible. > > That=E2=80=99s not the case of =E2=80=98tests/packages.scm=E2=80=99 thoug= h. Which gives me an > idea: what would it take to modify the test driver so it can print the > time spent in each test? :-) > > That would probably prove helpful to optimize core Guix. Good idea! I've pushed 5e652e94a9 that does this! Here's a sample, showing the three slowest test cases of the tests/packages.scm test: $ export SCM_LOG_DRIVER_FLAGS=3D"--brief=3Dno --show-duration=3Dyes" $ make check TESTS=3Dtests/packages.scm --8<---------------cut here---------------start------------->8--- [...] PASS: tests/packages.scm - fold-available-packages with/without cache [17.1= 21s] PASS: tests/packages.scm - find-packages-by-name [0.000s] PASS: tests/packages.scm - find-packages-by-name with version [0.000s] PASS: tests/packages.scm - find-packages-by-name with cache [9.492s] PASS: tests/packages.scm - find-packages-by-name + version, with cache [7.4= 12s] [...] PASS: tests/packages.scm - find-package-locations with cache [7.484s] PASS: tests/packages.scm - specification->location [0.000s] --8<---------------cut here---------------end--------------->8--- Automake could be extended to support this extra data; it would be interesting to be able to set the number of slowest tests to display, such as DURATIONS=3D5, to show only the 5 slowest tests, for example. > In many cases, I also run the one test I=E2=80=99m interested in through = Geiser, > which gives an optimally fast feedback loop. I've done that too, although it's somewhat clunky to use: the test modules name not being mapped to an actual filename hierarchy trips the module loading mechanism of Guile/Geiser. I usually fix that such that (test-packages) -> (tests packages), then I can load the module normally. Except that's not what I want to do as it runs all the tests. So I must copy paste the module declaration to the REPL to get the symbols imported, then the other required definitions and the (test-begin... due to SRFI 64 stateful design. Not my idea of a sleek test feedback loop, coming from pytest and other modern test frameworks; still very powerful to be able to do everything at the REPL though! >> We can also check the time the suspected slow test took: >> >> $ time make check TESTS=3Dtests/packages.scm >> SCM_LOG_DRIVER_FLAGS=3D"--select=3D'fold-available-packages with/without >> cache'" >> [...] >> PASS: tests/packages.scm - fold-available-packages with/without cache > > Ah ha! Turns out a large part of the time was due to the O(n=C2=B2) beha= vior > of the various list operations (the list of packages is big enough!). > Fixed in 73744725dd0a65cddaa9251f104f17ca27756479. Woohoo! That looks clever! Too clever for me to understand in the 2 minutes I starred at it ;-). That test now runs almost 6x faster (~8 s on a fast machine). Well done! >>>> +The underlying SRFI 64 custom Automake test driver used for the 'chec= k' >>>> +test suite (located at @file{build-aux/test-driver.scm}) also allows >>> >>> Maybe shorten to =E2=80=9CThe underlying test driver (located at >>> @file{build-aux/test-driver.scm}) also allows=E2=80=9D. >> >> I see value in explicitly stating what it is, as it took me some effort >> to be able to answer that question when I started looking at it (the >> test driver). > > Agreed. It just seemed to me that we were mentioning three new > concepts/tools in passing: SRFI-64, Automake, and test drivers. Yeah, it's a mouthful, for better or worse. Thank you, Maxim From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 02 03:47:09 2021 Received: (at 45948-done) by debbugs.gnu.org; 2 Feb 2021 08:47:09 +0000 Received: from localhost ([127.0.0.1]:34007 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6rL2-0007k0-Rz for submit@debbugs.gnu.org; Tue, 02 Feb 2021 03:47:09 -0500 Received: from mail-wm1-f53.google.com ([209.85.128.53]:53720) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6rL0-0007jC-Dk for 45948-done@debbugs.gnu.org; Tue, 02 Feb 2021 03:47:07 -0500 Received: by mail-wm1-f53.google.com with SMTP id j11so765072wmi.3 for <45948-done@debbugs.gnu.org>; Tue, 02 Feb 2021 00:47:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version:content-transfer-encoding; bh=swxB/+mOKi0eGMldzxHBkvpGCKsBG2SSSuoSOVGbhqg=; b=dwKqTUEGpfSKvvrXxRbwjRqw7MQTwWwaiQcH7RMkgKlGBh0jd14lUOzJ/A1dDHHWhQ BwRxSDbmKsmuen9ow7Pq1es7ammys2aSLcHAxayXJqZwaMf9FJXMopHFcpo9vH2KI9uR GasPeIJqZLosO5F63YCfzYmcJHEpDqbiExnpfbk9nRSR3GVn9WdA3OuvnRwR1ZA4Q43q vKXKWJgV9iEkX2XCHTw9yPgn1etcXcgzUqD/jWQfWUAdGbCBjXHw1xemdF6nSR4hmwp8 NIzK6C0+NYSOv1vKOnExOkDoLz+M5kAdNZFYq+oVoUF7a2MFluUgXv+AtTPKtnZRxlLj sqKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version:content-transfer-encoding; bh=swxB/+mOKi0eGMldzxHBkvpGCKsBG2SSSuoSOVGbhqg=; b=ANq9fCbIDzYn1QrL32Bzmb5v3Kmqz7V0P3umj3EN18BcZ4LatqgD6acbC3DwBJ2ICw NfBe2u6hLnPULAYe61EFX15DyXwE8HH2sIhJTv7b8ocqijZNUupFmEvvDzjJZD6stBIU B/npIHHsQxLiIDnfN61sLdNwmPkij25ueCdb1TexYDD+iTmeK/6FePQ4uFOH87G4VoO0 POacHxsRgokTd+3UOU5iG3xkX74wNmb+ZmHF92CKCHa4oX1bnZqbK0h/3XeR0gwYue9d c1Es4r2MU2btQ0lUTjgEmUIgSRN/SJeeDLJwiq6PEpDB64N0qaU82hVoPhP6yLp2wpS4 Xh/w== X-Gm-Message-State: AOAM532TUh98g+aBVvUoxkw9tFscj7iC79WOFwAbKAvwE5/OO/w/rW2T Ar+NboI7Om18gcmQYoB1xPIZlrrSyr8= X-Google-Smtp-Source: ABdhPJzkIpFexEVYorS18RXPX7Z/UEL3TRS2BnYsTu4xFGUJ43C75dD2jHO0uBWvEBTbqsi1QOPeZg== X-Received: by 2002:a7b:cb86:: with SMTP id m6mr347357wmi.45.1612255620256; Tue, 02 Feb 2021 00:47:00 -0800 (PST) Received: from lili ([2a01:e0a:59b:9120:65d2:2476:f637:db1e]) by smtp.gmail.com with ESMTPSA id r16sm9160935wrt.68.2021.02.02.00.46.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Feb 2021 00:46:59 -0800 (PST) From: zimoun To: Maxim Cournoyer , Ludovic =?utf-8?Q?Court?= =?utf-8?Q?=C3=A8s?= Subject: Re: [bug#45948] [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. In-Reply-To: <87h7mvqaw0.fsf@gmail.com> References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> <87eei2p0qu.fsf_-_@gnu.org> <87lfc8sb8w.fsf@gmail.com> <87czxjh1os.fsf@inria.fr> <87h7mvqaw0.fsf@gmail.com> Date: Tue, 02 Feb 2021 09:36:11 +0100 Message-ID: <86h7mudfyc.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948-done Cc: 45948-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, On Tue, 02 Feb 2021 at 00:47, Maxim Cournoyer w= rote: > $ export SCM_LOG_DRIVER_FLAGS=3D"--brief=3Dno --show-duration=3Dyes" > $ make check TESTS=3Dtests/packages.scm > > --8<---------------cut here---------------start------------->8--- > [...] > PASS: tests/packages.scm - fold-available-packages with/without cache [17= .121s] > PASS: tests/packages.scm - find-packages-by-name [0.000s] > PASS: tests/packages.scm - find-packages-by-name with version [0.000s] > PASS: tests/packages.scm - find-packages-by-name with cache [9.492s] > PASS: tests/packages.scm - find-packages-by-name + version, with cache [7= .412s] > [...] > PASS: tests/packages.scm - find-package-locations with cache [7.484s] > PASS: tests/packages.scm - specification->location [0.000s] > --8<---------------cut here---------------end--------------->8--- Cool! Thanks Maxim. I find it really better. :-) >>> We can also check the time the suspected slow test took: >>> >>> $ time make check TESTS=3Dtests/packages.scm >>> SCM_LOG_DRIVER_FLAGS=3D"--select=3D'fold-available-packages with/without >>> cache'" >>> [...] >>> PASS: tests/packages.scm - fold-available-packages with/without cache >> >> Ah ha! Turns out a large part of the time was due to the O(n=C2=B2) beh= avior >> of the various list operations (the list of packages is big enough!). >> Fixed in 73744725dd0a65cddaa9251f104f17ca27756479. Neat! Cool! Cheers, simon From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 02 07:53:10 2021 Received: (at 45948-done) by debbugs.gnu.org; 2 Feb 2021 12:53:10 +0000 Received: from localhost ([127.0.0.1]:34302 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6vB7-0005mj-Q4 for submit@debbugs.gnu.org; Tue, 02 Feb 2021 07:53:10 -0500 Received: from mail-qv1-f42.google.com ([209.85.219.42]:45173) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l6vB4-0005mG-TC for 45948-done@debbugs.gnu.org; Tue, 02 Feb 2021 07:53:08 -0500 Received: by mail-qv1-f42.google.com with SMTP id w11so9762409qvz.12 for <45948-done@debbugs.gnu.org>; Tue, 02 Feb 2021 04:53:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=lQuMKsgn9Tjj40jxO0xxwczSFIsQ4cfBxLLCGfvil0g=; b=ZbsTE2rZEKu10ZWCLDwAtMeRZEjK8WaTtG2VZSqWmzpJ4CxA7mcb2aPnFE27oLNtvi ewSrS7tpeohsPuECxxlrQVgKYc1qrUKTAX8wCoM6ilBgyeRcQytwkHeb1c6Nx1cVJduU nroJMCmS0sGhRa2EIb3z63Xn2OeQ8fjrUzKgzT/WPko18WqSBfDqDguZubE5bB8REcE8 JwvrdOwVrN1odOPQKbCijx5ORw9ojXvLhjU5sBhK9eYwJE6lbG2goJFT+tthXmCsO2em 9bXpqXLpdSXZ2pvAUxE2R4jD+84WfYDIi3CBFKp9XqDzSPyqAf1Gj7Tk79uzSTXC42I8 610g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=lQuMKsgn9Tjj40jxO0xxwczSFIsQ4cfBxLLCGfvil0g=; b=esW8ufl6kKzIcDSYae860nisiC2z6f3zwiCpoEni9XezYnKSybPFkD+qyyR8Vj+UfV JtbVh5sMCCoKsFJFw1hAQx2LI0yc5Yf+YlH1pvkVZ9Q2sjBEg/vhvtzgeNOaiWe70q15 3PgoQ5CcvadKXwk8LL/OKuOrtQRmd3VP+ylg4MB/3Lp1h4o2YRMmIVDw5ud04YxLfIqj IWEk0teh/gl69B0Yuz6ne+sCSsHq3pNdUl8IwozBsHjCB7rj6qlpnzkmBmdDEzqbFL/N L9m7IxrWmFetE9bEu77Rl2Hlx+Hb+Kc4ZBcP4I/QeEZ3e4K5RclclCvenTKFaRS3KsDQ xNMw== X-Gm-Message-State: AOAM530IHe6W3rRPbB8W1tXDiQhbdU5OnkDF9c+keUs83oxZ3ew5zxFk PRGT/mZAb56OK3SlEM/1KnOs6rxBjbQGaA== X-Google-Smtp-Source: ABdhPJw9bMm5+hyd713RPs2JbTRDDEymeHnnO9QMNrjkQxcxDiZzhi+50zZE1Qqk3fJrY1hSl4CpjA== X-Received: by 2002:a0c:f844:: with SMTP id g4mr10168473qvo.60.1612270381093; Tue, 02 Feb 2021 04:53:01 -0800 (PST) Received: from hurd (dsl-157-110.b2b2c.ca. [66.158.157.110]) by smtp.gmail.com with ESMTPSA id 2sm15876493qtt.24.2021.02.02.04.52.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Feb 2021 04:53:00 -0800 (PST) From: Maxim Cournoyer To: zimoun Subject: Re: [bug#45948] [PATCH 0/5] Improvements to the Automake SRFI 64 test driver. References: <20210118062501.27022-1-maxim.cournoyer@gmail.com> <20210118062501.27022-5-maxim.cournoyer@gmail.com> <87eei2p0qu.fsf_-_@gnu.org> <87lfc8sb8w.fsf@gmail.com> <87czxjh1os.fsf@inria.fr> <87h7mvqaw0.fsf@gmail.com> <86h7mudfyc.fsf@gmail.com> Date: Tue, 02 Feb 2021 07:52:59 -0500 In-Reply-To: <86h7mudfyc.fsf@gmail.com> (zimoun's message of "Tue, 02 Feb 2021 09:36:11 +0100") Message-ID: <87czxir5qs.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45948-done Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 45948-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello Simon! zimoun writes: > Hi, > > On Tue, 02 Feb 2021 at 00:47, Maxim Cournoyer wrote: > >> $ export SCM_LOG_DRIVER_FLAGS="--brief=no --show-duration=yes" >> $ make check TESTS=tests/packages.scm >> >> --8<---------------cut here---------------start------------->8--- >> [...] >> PASS: tests/packages.scm - fold-available-packages with/without cache [17.121s] >> PASS: tests/packages.scm - find-packages-by-name [0.000s] >> PASS: tests/packages.scm - find-packages-by-name with version [0.000s] >> PASS: tests/packages.scm - find-packages-by-name with cache [9.492s] >> PASS: tests/packages.scm - find-packages-by-name + version, with cache [7.412s] >> [...] >> PASS: tests/packages.scm - find-package-locations with cache [7.484s] >> PASS: tests/packages.scm - specification->location [0.000s] >> --8<---------------cut here---------------end--------------->8--- > > Cool! Thanks Maxim. I find it really better. :-) I'm glad you find these changes useful! Cheers, Maxim From unknown Fri Aug 15 02:03:41 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 03 Mar 2021 12:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator