Subject: [ERT] EMACS_TEST_VERBOSE can not be explicitly disabled

From: Max Nikulin <[email-masked]>
Date: Mon, 17 Jan 2022 17:59:33 +0700

At least empty value of EMACS_TEST_VERBOSE environment variable
should be considered as false value that disables verbose ERT summary.

Currently ERT implementation does not distinguish non-empty and empty
values of the variable and generates verbose summary when such
variable is just present in the process environment.

To improve user experience some strings widely used in configuration 
files as false values should be recognized by ERT as well. Examples 
(case insensitive): "0", "false", "N", "no", "off", "none", "nil".

Recently Ihor Radchenko proposed to enable verbose summary mode by 
default for Org Mode tests when it is built as a standalone project 
outside of Emacs source tree:
https://lists.gnu.org/archive/html/emacs-orgmode/2022-01/msg00022.html
I like the idea, but I can not say the same concerning implementation. 
Ihor introduced new make variable while I believe that it is not 
necessary and Org makefiles should be transparent in relation to 
EMACS_TEST_VERBOSE.

I expect that it should be enough to add
     EMACS_TEST_VERBOSE ?= yes
     export EMACS_TEST_VERBOSE
to the default.mk file
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/mk/default.mk

Actually with current implementation developer has no chance to disable
verbose summary mode from command line since empty string is considered
as true value by ERT implementation. The following does not work:

     EMACS_TEST_VERBOSE= make test
     make test EMACS_TEST_VERBOSE=

A workaround is "unexport EMACS_TEST_VERBOSE" either in the local.mk
file or wrapped with ifeq-else-endif in default.mk.

Another case is that a developer may have
"export EMACS_TEST_VERBOSE=yes" in their shell init file
but sometimes wants to disable verbose summary for a couple of commands
without changing of environment by "unset EMACS_TEST_VERBOSE"
that will affect following commands. The following commands
should cause different states of verbose summary setting
independently of current environment:

     make test EMACS_TEST_VERBOSE=
     make test EMACS_TEST_VERBOSE=yes

Standalone example: ert-sample.el
---- >8 ----
(require 'ert)
(ert-deftest ert-sample ()
   (should (equal 5 (* 2 2))))
(ert-run-tests-batch-and-exit)
---- 8< ----

     export EMACS_TEST_VERBOSE=yes  # in ~/.profile, ~/.bashrc, etc.
     EMACS_TEST_VERBOSE= emacs --batch -Q ert-sample.el

I have not found any discussion of design choices related to this variable.

To wrap up, I suppose that at least empty value of EMACS_TEST_VERBOSE
should be considered as a request to generate brief summary.
It would be great to have opportunity to explicitly say "no":
EMACS_TEST_VERBOSE=off (or "false", "0", etc.).
It will make it easier for various build tools to respect
developer's choice.