GNU bug report logs -
#72536
[PATCH] tap: new `--merge-prefix' option to prefix stderr before merging
Previous Next
Reported by: Richard Hansen <rhansen <at> rhansen.org>
Date: Fri, 9 Aug 2024 08:40:01 UTC
Severity: normal
Tags: patch
Done: Karl Berry <karl <at> freefriends.org>
Bug is archived. No further changes may be made.
Full log
View this message in rfc822 format
[Message part 1 (text/plain, inline)]
Your message dated Sun, 23 Feb 2025 10:59:32 -0700
with message-id <202502231759.51NHxWtR994843 <at> freefriends.org>
and subject line Re: [PATCH v3] tap: new `--stderr-prefix' option to prefix test script's stderr
has caused the debbugs.gnu.org bug report #72536,
regarding [PATCH] tap: new `--merge-prefix' option to prefix stderr before merging
to be marked as done.
(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)
--
72536: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=72536
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems
[Message part 2 (message/rfc822, inline)]
[Message part 3 (text/plain, inline)]
Attached are a couple of patches to improve the TAP driver. I don't know if `--merge-prefix' is the best name for the new option; suggestions for alternatives would be appreciated.
Thanks,
Richard
[PATCH 1/2] tap: new `--merge-prefix' option to prefix stderr before merging
* lib/tap-driver.sh: Add a new `--merge-prefix' option to direct the TAP
driver to prefix each line of the test script's standard error with the given
string before merging it with the test script's standard output. This is
useful when standard error lines might be confused with test results.
* doc/automake.texi: Document the new `--merge-prefix' option.
* t/tap-merge-prefix.sh: New test.
* t/list-of-tests.mk (handwritten_TESTS): Add it.
[PATCH 2/2] tap: close fd 3 when invoking the test script
* lib/tap-driver.sh: Close file descriptor 3 when invoking the test script to
avoid potential conflicts with the test script.
[0001-tap-new-merge-prefix-option-to-prefix-stderr-before-.patch (text/x-patch, attachment)]
[0002-tap-close-fd-3-when-invoking-the-test-script.patch (text/x-patch, attachment)]
[OpenPGP_signature.asc (application/pgp-signature, attachment)]
[Message part 7 (message/rfc822, inline)]
Hi Richard - here's my tinkering with the description in the manual
(pushed). Let me know if you see any problems.
Meanwhile, I guess we can close this. If anything more is pending from
your side, let me know or feel free to re-open. Thanks again. -k
-----------------------------------------------------------------------------
doc: move most --stderr-prefix doc to a new subsubsection.
Still in https://bugs.gnu.org/72536.
* doc/automake.texi (TAP setup and examples):
(TAP stderr examples),
TAP stderr prefixing): split TAP examples
into new subsubsections. Add discussion of race condition and
other reordering from the open bug.
* NEWS: mention --stderr-prefix.
diff --git a/NEWS b/NEWS
index 34876f3db..26b433314 100644
--- a/NEWS
+++ b/NEWS
@@ -6,6 +6,9 @@ New in 1.x:
* New features added
+ - New option --stderr-prefix for tap-driver.sh, to prefix each line of
+ stderr from a test script with a given string. (bug#72536)
+
- Support for Algol 68 added, based on the GNU Algol 68 compiler. (bug#75807)
- New option dist-bzip3 for bzip3 compression of distributions. (bug#73795)
@@ -18,6 +21,9 @@ New in 1.x:
- Avoid Perl 5.41.8+ precedence warning for use of !!.
(https://lists.gnu.org/archive/html/automake/2025-01/msg00000.html)
+ - a Perl path containing whitespace now emits a warning instead of
+ an error, so ./configure PERL='/usr/bin/env perl' can work. (bug#74453)
+
- The py-compile script once again does nothing (successfully) if the
PYTHON environment variable is set to ":", or anything that isn't a
Python interpreter (according to $PYTHON -V). Exception: if PYTHON
diff --git a/doc/automake.texi b/doc/automake.texi
index 9fa94689f..bb66aea5c 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -10516,6 +10516,7 @@ with the @samp{#} character) in the testsuite progress output too; by
default, TAP diagnostics are only copied to the @file{.log} file.
@item --no-comments
Revert the effects of @option{--comments}.
+
@item --merge
Instruct the test driver to merge the test scripts' standard error into
their standard output. This is necessary if you want to ensure that
@@ -10526,8 +10527,10 @@ tracing active). As a downside, this option might cause the test
harness to get confused if anything that appears on standard error
looks like a test result. Use @option{--stderr-prefix} to prefix each
line of standard error to avoid this problem.
+
@item --no-merge
Revert the effects of @option{--merge}.
+
@item --stderr-prefix @var{STRING}
Prefix each line of the test script's standard error with
@code{@var{STRING}}. Defaults to the empty string. This option makes
@@ -10549,39 +10552,13 @@ literal characters @samp{@@%:@@} in the prefix (@samp{@@%@@&t@@:@@}, for
example).
@end table
-For example, in shell, @samp{--stderr-prefix "@@%:@@[stderr] "} will
-prefix each line of the test script's standard error with
-@samp{#[stderr] }.
-
-When prefixing standard error lines, the test program's standard error
-lines can be processed out of order relative to the test program's
-standard output lines, even when @option{--merge} is used. This can be
-due to many factors, including:
-
-@itemize @bullet
-@item
-Output buffering in the test program: The test program might be
-accumulating output data and writing it to the operating system in
-batches, not as soon as it is available. You can try
-@uref{man:setvbuf(3),,setvbuf(3)} to change the buffering mode in the
-test program, or you can add regular flushes. OS-specific utilities to
-control buffering may be available, such as
-@uref{man:stdbuf(1),,stdbuf(1)} on GNU/Linux systems.
+For example, the option @samp{--stderr-prefix "@@%:@@[stderr] "}
+passed on the command line will prefix each line of the test script's
+standard error with @samp{#[stderr] }.
-@item
-Concurrent processing of standard output and standard error: Due to a
-limitation of the shell command language, the TAP driver processes
-standard output in a different thread or process than standard error.
-Thus, processing order is influenced by how the operating system
-schedules the two threads or processes for execution.
-
-@item
-Line-based processing: When prefixing standard error, the TAP driver
-waits until a complete line is read from the test program before
-processing the line. This reduces the chances that characters from a
-standard output line are mixed with characters from a standard error
-line.
-@end itemize
+For more details on @option{--stderr-prefix} processing, especially
+the occasional output reordering due to a race condition, @pxref{TAP
+prefixing stderr}.
@item --diagnostic-string @var{STRING}
Change the string that introduces TAP diagnostics from the default value
@@ -10596,7 +10573,18 @@ the ``official'' TAP protocol does not allow for such a customization; so
don't use it if you can avoid it.
@end table
-@noindent
+@menu
+* TAP setup and examples:: Setup and examples of TAP usage.
+* TAP stderr examples:: Examples of stderr issues and processing.
+* TAP prefixing stderr:: Details about @option{--stderr-prefix} processing.
+@end menu
+
+@node TAP setup and examples
+@subsubsection TAP setup and examples
+
+@cindex TAP setup, example of
+@cindex setting up TAP, example
+
Here is an example of how the TAP driver can be set up and used.
@c Keep in sync with tap-doc2.sh
@@ -10670,7 +10658,19 @@ PASS: baz.test 1
...
% @kbd{echo exit status: $?}
exit status: 0
+@end example
+@node TAP stderr examples
+@subsubsection TAP stderr examples
+
+@cindex TAP stderr example
+@cindex stderr example in TAP
+
+Here is a sequence of examples showing the issue with merging stdout
+and stderr, and the resolution with @option{--stderr-prefix}. Read
+the comments for specifics.
+
+@example
% @kbd{cat stderr-demo.test}
#!/bin/sh
echo 'ok 1'
@@ -10721,9 +10721,69 @@ PASS: stderr-demo.test 1
ok 2
PASS: stderr-demo.test 2
1..2
-
@end example
+@node TAP prefixing stderr
+@subsubsection TAP prefixing stderr
+
+@cindex TAP prefixing stderr
+@cindex prefixing stderr in TAP
+@cindex stderr prefixing in TAP
+@cindex race condition in stderr prefixing
+
+@c see https://bugs.gnu.org/72536 for this feature and discussion.
+
+When using the above @option{--stderr-prefix} TAP option, the test
+program's standard error lines might occasionally be processed out of
+order relative to the test program's standard output lines,
+particularly when @option{--merge} is also used. This can be due to
+many factors, including:
+
+@itemize @bullet
+@item
+Output buffering in the test program: The test program might be
+accumulating output data and writing it to the operating system in
+batches, not as soon as it is available. You can try setvbuf(3) to
+change the buffering mode in the test program, or you can add regular
+flushes. OS-specific utilities to control buffering may be available,
+such as stdbuf(1) on GNU/Linux systems.
+
+@item
+Concurrent processing of standard output and standard error: Due to a
+limitation of the shell command language, the TAP driver processes
+standard output in a different thread or process than standard error,
+causing a race condition. Thus, processing order is influenced by how
+the operating system schedules the two threads or processes for
+execution.
+
+@item
+Line-based processing: When prefixing standard error, the TAP driver
+waits until a complete line is read from the test program before
+processing the line. This reduces the chances that characters from a
+standard output line are mixed with characters from a standard error
+line.
+@end itemize
+
+Thus, stderr lines might appear out of order relative to stdout lines,
+even when @option{--merge} is passed. All stderr lines are still
+processed in order relative to other stderr lines, and the same goes
+for the stdout lines relative to other stdout lines. So the
+reordering shouldn't break a package's TAP tests, since there's no
+reason to output TAP commands to stderr; even more so when stderr
+lines are prefixed.
+
+When working on a particular test, if you find the reordering is
+causing trouble, you can omit @option{--stderr-prefix}, or run the
+test directly (not via TAP) to get a more accurate understanding of
+output timing.
+
+Overall, though, the order of processing (and logging) should not
+differ from the order of the test program's write(2) calls by much.
+Out-of-order processing only happens if the test program writes to
+stdout then stderr but the kernel wakes up the TAP driver's
+thread/process that is reading stderr before it wakes up the TAP
+driver's thread/process that is reading stdout, or vice versa.
+
@node Incompatibility with other TAP parsers and drivers
@subsection Incompatibility with other TAP parsers and drivers
compile finished at Sun Feb 23 09:55:43 2025
This bug report was last modified 96 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.