Hi Jacob,

Thank you for your quick response!

You are welcome.

Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a thanks in the ChangeLog?

Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer <jcb62281@gmail.com>:

On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:

Hi,

The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
check the exit status of the executed test programs. A test program
that simply aborts (e.g. using the C function of the same name) won't
cause any testsuite failures. This seems brittle and like a
misfeature.

What is the supposed way to deal with this?

The host_execute procedure is intended for running unit test programs
that speak a special DejaGnu protocol and ignoring exit codes from unit
test programs is intended.  The DejaGnu unit testing protocol does not
depend on the exit code of the unit test program at all, because that
exit code might not be available in all environments.

I am sorry if I wasn't clear enough in my previous email. I didn't
mean that the exit code should be part of the actual protocol, only
that running the testcase should be considered unsuccessful if the
program exits with a failure code (in a POSIX system; in some other
environment, there may be other indications for failure of execution).

But that would make the exit code part of the protocol.  DejaGnu generally supports running tests on "remote" target boards and a target connected over a serial line is unlikely to return an exit code.  The only way to be sure that a dependency on the exit code will not creep in is to ignore the exit code.

Admittedly, DejaGnu does not yet properly support running unit tests on remote targets.

Instead, DejaGnu expects an explicit "END" token from the unit test
program to indicate that the program has reached its intended
completion.  A warning is produced if this token is not observed;
perhaps a future version of DejaGnu should insert an UNRESOLVED result
like we currently do when a Tcl test script aborts?

I think inserting an UNRESOLVED result would be more appropriate.
Otherwise, there would be no formal failure result if a unit test
breaks due to, say, a segfault.

The explicit "END" token is a relatively recent addition (added after the last release).  I was reluctant to outright require it (in case there are any testsuites out there with independent unit test protocol implementations) but I now agree that a warning from the test framework is not enough when a unit test bombs out early.

An initial solution has been pushed to Savannah on the PR79077 branch. DejaGnu can now be run directly from a Git checkout, you should be able to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make check" command line.

libgccjit uses a heavily patched version of host_execute ([1]) that
also handles tests run under Valgrind; perhaps some of the ideas from
that version can be incorporated into DejaGnu proper.

That heavily patched version has problems that have since been fixed upstream.  I would encourage you to try getting rid of it.

You should be able to move the Valgrind support into a wrapper around host_execute, since host_execute has accepted arguments since 1.6.3; you can test precisely by matching [info body host_execute] against "The arguments are".  Note that the name of the executable needs to be absolute; the DejaGnu utility proc "which" will search PATH for its argument.

A first draft (untested) off the top of my head:

proc wrap_host_execute {args} {
    global env

    set run_under_valgrind [info exists env(RUN_UNDER_VALGRIND)]

    set exec_args $args
    set executable [lindex $args 0]
    set arguments [lrange $args 1 end]

    if { $run_under_valgrind } {
	set valgrind_logfile "${executable}.valgrind.txt"
	set valgrind_params [which "valgrind"]
	lappend valgrind_params "--leak-check=full"
	lappend valgrind_params "--log-file=${valgrind_logfile}"
	set exec_args [eval [list linsert $exec_args 0] $valgrind_params]
    }

    set reported_error [eval [list host_execute] $exec_args]

    if { $run_under_valgrind } {
	upvar 2 name name
	parse_valgrind_logfile $name $valgrind_logfile xfail
    }

    return $reported_error
}

You should probably instead use a VALGRIND global and have one of the initialization files default it to [which "valgrind"] if RUN_UNDER_VALGRIND is set in the environment.  I would recommend taking VALGRIND from the first of:

    (1)  value given on DejaGnu command line ([info exists VALGRIND])
    (2)  environment variable VALGRIND ($::env(VALGRIND) in Tcl)
    (3)  default by searching PATH ([which "valgrind"])

A first draft (untested) for that initialization code off the top of my head:

if { ![info exists VALGRIND] && [info exists ::env(RUN_UNDER_VALGRIND)] } {
    if { [info exists ::env(VALGRIND)] } {
	set VALGRIND $::env(VALGRIND)
    } else {
	set VALGRIND [which "valgrind"]
    }
}

The code does nothing if VALGRIND is already set because the framework will set it as a Tcl global if it is given on the command line.  A framework-provided value can also originate from multi-pass testing and vary between passes, perhaps to compare results from different Valgrind versions.

[1] https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89

-- Jacob