Package: dejagnu;
Reported by: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
Date: Wed, 23 Jul 2025 07:33:01 UTC
Severity: normal
Done: Jacob Bachmeyer <jcb62281 <at> gmail.com>
View this message in rfc822 format
From: Jacob Bachmeyer <jcb62281 <at> gmail.com> To: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com> Cc: 79077 <at> debbugs.gnu.org Subject: bug#79077: host_execute and non-zero exit status Date: Thu, 24 Jul 2025 21:13:53 -0500
[Message part 1 (text/plain, inline)]
On 7/24/25 00:57, Marc Nieper-Wißkirchen wrote:
> Hi Jacob,
>
> Thank you for your quick response!
You are welcome.
Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a
thanks in the ChangeLog?
> Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer<jcb62281 <at> gmail.com>:
>> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
>>> Hi,
>>>
>>> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
>>> check the exit status of the executed test programs. A test program
>>> that simply aborts (e.g. using the C function of the same name) won't
>>> cause any testsuite failures. This seems brittle and like a
>>> misfeature.
>>>
>>> What is the supposed way to deal with this?
>> The host_execute procedure is intended for running unit test programs
>> that speak a special DejaGnu protocol and ignoring exit codes from unit
>> test programs is intended. The DejaGnu unit testing protocol does not
>> depend on the exit code of the unit test program at all, because that
>> exit code might not be available in all environments.
> I am sorry if I wasn't clear enough in my previous email. I didn't
> mean that the exit code should be part of the actual protocol, only
> that running the testcase should be considered unsuccessful if the
> program exits with a failure code (in a POSIX system; in some other
> environment, there may be other indications for failure of execution).
But that would make the exit code part of the protocol. DejaGnu
generally supports running tests on "remote" target boards and a target
connected over a serial line is unlikely to return an exit code. The
only way to be sure that a dependency on the exit code will not creep in
is to ignore the exit code.
Admittedly, DejaGnu does not yet properly support running unit tests on
remote targets.
>> Instead, DejaGnu expects an explicit "END" token from the unit test
>> program to indicate that the program has reached its intended
>> completion. A warning is produced if this token is not observed;
>> perhaps a future version of DejaGnu should insert an UNRESOLVED result
>> like we currently do when a Tcl test script aborts?
> I think inserting an UNRESOLVED result would be more appropriate.
> Otherwise, there would be no formal failure result if a unit test
> breaks due to, say, a segfault.
The explicit "END" token is a relatively recent addition (added after
the last release). I was reluctant to outright require it (in case
there are any testsuites out there with independent unit test protocol
implementations) but I now agree that a warning from the test framework
is not enough when a unit test bombs out early.
An initial solution has been pushed to Savannah on the PR79077 branch.
DejaGnu can now be run directly from a Git checkout, you should be able
to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make
check" command line.
> libgccjit uses a heavily patched version of host_execute ([1]) that
> also handles tests run under Valgrind; perhaps some of the ideas from
> that version can be incorporated into DejaGnu proper.
That heavily patched version has problems that have since been fixed
upstream. I would encourage you to try getting rid of it.
You should be able to move the Valgrind support into a wrapper around
host_execute, since host_execute has accepted arguments since 1.6.3; you
can test precisely by matching [info body host_execute] against "The
arguments are". Note that the name of the executable needs to be
absolute; the DejaGnu utility proc "which" will search PATH for its
argument.
A first draft (untested) off the top of my head:
proc wrap_host_execute {args} {
global env
set run_under_valgrind [info exists env(RUN_UNDER_VALGRIND)]
set exec_args $args
set executable [lindex $args 0]
set arguments [lrange $args 1 end]
if { $run_under_valgrind } {
set valgrind_logfile "${executable}.valgrind.txt"
set valgrind_params [which "valgrind"]
lappend valgrind_params "--leak-check=full"
lappend valgrind_params "--log-file=${valgrind_logfile}"
set exec_args [eval [list linsert $exec_args 0] $valgrind_params]
}
set reported_error [eval [list host_execute] $exec_args]
if { $run_under_valgrind } {
upvar 2 name name
parse_valgrind_logfile $name $valgrind_logfile xfail
}
return $reported_error
}
You should probably instead use a VALGRIND global and have one of the
initialization files default it to [which "valgrind"] if
RUN_UNDER_VALGRIND is set in the environment. I would recommend taking
VALGRIND from the first of:
(1) value given on DejaGnu command line ([info exists VALGRIND])
(2) environment variable VALGRIND ($::env(VALGRIND) in Tcl)
(3) default by searching PATH ([which "valgrind"])
A first draft (untested) for that initialization code off the top of my
head:
if { ![info exists VALGRIND] && [info exists ::env(RUN_UNDER_VALGRIND)] } {
if { [info exists ::env(VALGRIND)] } {
set VALGRIND $::env(VALGRIND)
} else {
set VALGRIND [which "valgrind"]
}
}
The code does nothing if VALGRIND is already set because the framework
will set it as a Tcl global if it is given on the command line. A
framework-provided value can also originate from multi-pass testing and
vary between passes, perhaps to compare results from different Valgrind
versions.
> [1]https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89
-- Jacob
[Message part 2 (text/html, inline)]
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.