Subject: bug#79077: host_execute and non-zero exit status

From: Jacob Bachmeyer <[email-masked]>
Date: Thu, 24 Jul 2025 21:13:53 -0500

On 7/24/25 00:57, Marc Nieper-Wißkirchen wrote:
> Hi Jacob,
>
> Thank you for your quick response!

You are welcome.

Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a 
thanks in the ChangeLog?

> Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer<[email-masked]>:
>> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
>>> Hi,
>>>
>>> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
>>> check the exit status of the executed test programs. A test program
>>> that simply aborts (e.g. using the C function of the same name) won't
>>> cause any testsuite failures. This seems brittle and like a
>>> misfeature.
>>>
>>> What is the supposed way to deal with this?
>> The host_execute procedure is intended for running unit test programs
>> that speak a special DejaGnu protocol and ignoring exit codes from unit
>> test programs is intended.  The DejaGnu unit testing protocol does not
>> depend on the exit code of the unit test program at all, because that
>> exit code might not be available in all environments.
> I am sorry if I wasn't clear enough in my previous email. I didn't
> mean that the exit code should be part of the actual protocol, only
> that running the testcase should be considered unsuccessful if the
> program exits with a failure code (in a POSIX system; in some other
> environment, there may be other indications for failure of execution).

But that would make the exit code part of the protocol.  DejaGnu 
generally supports running tests on "remote" target boards and a target 
connected over a serial line is unlikely to return an exit code.  The 
only way to be sure that a dependency on the exit code will not creep in 
is to ignore the exit code.

Admittedly, DejaGnu does not yet properly support running unit tests on 
remote targets.

>> Instead, DejaGnu expects an explicit "END" token from the unit test
>> program to indicate that the program has reached its intended
>> completion.  A warning is produced if this token is not observed;
>> perhaps a future version of DejaGnu should insert an UNRESOLVED result
>> like we currently do when a Tcl test script aborts?
> I think inserting an UNRESOLVED result would be more appropriate.
> Otherwise, there would be no formal failure result if a unit test
> breaks due to, say, a segfault.

The explicit "END" token is a relatively recent addition (added after 
the last release).  I was reluctant to outright require it (in case 
there are any testsuites out there with independent unit test protocol 
implementations) but I now agree that a warning from the test framework 
is not enough when a unit test bombs out early.

An initial solution has been pushed to Savannah on the PR79077 branch. 
DejaGnu can now be run directly from a Git checkout, you should be able 
to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make 
check" command line.

> libgccjit uses a heavily patched version of host_execute ([1]) that
> also handles tests run under Valgrind; perhaps some of the ideas from
> that version can be incorporated into DejaGnu proper.

That heavily patched version has problems that have since been fixed 
upstream.  I would encourage you to try getting rid of it.

You should be able to move the Valgrind support into a wrapper around 
host_execute, since host_execute has accepted arguments since 1.6.3; you 
can test precisely by matching [info body host_execute] against "The 
arguments are".  Note that the name of the executable needs to be 
absolute; the DejaGnu utility proc "which" will search PATH for its 
argument.

A first draft (untested) off the top of my head:

proc wrap_host_execute {args} {
     global env

     set run_under_valgrind [info exists env(RUN_UNDER_VALGRIND)]

     set exec_args $args
     set executable [lindex $args 0]
     set arguments [lrange $args 1 end]

     if { $run_under_valgrind } {
	set valgrind_logfile "${executable}.valgrind.txt"
	set valgrind_params [which "valgrind"]
	lappend valgrind_params "--leak-check=full"
	lappend valgrind_params "--log-file=${valgrind_logfile}"
	set exec_args [eval [list linsert $exec_args 0] $valgrind_params]
     }

     set reported_error [eval [list host_execute] $exec_args]

     if { $run_under_valgrind } {
	upvar 2 name name
	parse_valgrind_logfile $name $valgrind_logfile xfail
     }

     return $reported_error
}

You should probably instead use a VALGRIND global and have one of the 
initialization files default it to [which "valgrind"] if 
RUN_UNDER_VALGRIND is set in the environment.  I would recommend taking 
VALGRIND from the first of:

     (1)  value given on DejaGnu command line ([info exists VALGRIND])
     (2)  environment variable VALGRIND ($::env(VALGRIND) in Tcl)
     (3)  default by searching PATH ([which "valgrind"])

A first draft (untested) for that initialization code off the top of my 
head:

if { ![info exists VALGRIND] && [info exists ::env(RUN_UNDER_VALGRIND)] } {
     if { [info exists ::env(VALGRIND)] } {
	set VALGRIND $::env(VALGRIND)
     } else {
	set VALGRIND [which "valgrind"]
     }
}

The code does nothing if VALGRIND is already set because the framework 
will set it as a Tcl global if it is given on the command line.  A 
framework-provided value can also originate from multi-pass testing and 
vary between passes, perhaps to compare results from different Valgrind 
versions.

> [1]https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89

-- Jacob