GNU bug report logs - #79077
host_execute and non-zero exit status

Previous Next

To reply to this bug, email your comments to 79077 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: bug-dejagnu <at> gnu.org
Subject: host_execute and non-zero exit status
Date: Wed, 23 Jul 2025 09:31:21 +0200

Hi,

The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
check the exit status of the executed test programs. A test program
that simply aborts (e.g. using the C function of the same name) won't
cause any testsuite failures. This seems brittle and like a
misfeature.

What is the supposed way to deal with this?

Thanks,

Marc

--

[1] https://www.gnu.org/software/dejagnu/manual/Running-unit-tests.html

Message #8 received at 79077 <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 79077 <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Wed, 23 Jul 2025 22:08:43 -0500

On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
> Hi,
>
> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
> check the exit status of the executed test programs. A test program
> that simply aborts (e.g. using the C function of the same name) won't
> cause any testsuite failures. This seems brittle and like a
> misfeature.
>
> What is the supposed way to deal with this?

The host_execute procedure is intended for running unit test programs 
that speak a special DejaGnu protocol and ignoring exit codes from unit 
test programs is intended.  The DejaGnu unit testing protocol does not 
depend on the exit code of the unit test program at all, because that 
exit code might not be available in all environments.

Instead, DejaGnu expects an explicit "END" token from the unit test 
program to indicate that the program has reached its intended 
completion.  A warning is produced if this token is not observed; 
perhaps a future version of DejaGnu should insert an UNRESOLVED result 
like we currently do when a Tcl test script aborts?


-- Jacob

Message #11 received at 79077 <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: jcb62281 <at> gmail.com
Cc: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 79077 <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Thu, 24 Jul 2025 07:57:31 +0200

Hi Jacob,

Thank you for your quick response!

Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer <jcb62281 <at> gmail.com>:
>
> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
> > Hi,
> >
> > The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
> > check the exit status of the executed test programs. A test program
> > that simply aborts (e.g. using the C function of the same name) won't
> > cause any testsuite failures. This seems brittle and like a
> > misfeature.
> >
> > What is the supposed way to deal with this?
>
> The host_execute procedure is intended for running unit test programs
> that speak a special DejaGnu protocol and ignoring exit codes from unit
> test programs is intended.  The DejaGnu unit testing protocol does not
> depend on the exit code of the unit test program at all, because that
> exit code might not be available in all environments.

I am sorry if I wasn't clear enough in my previous email. I didn't
mean that the exit code should be part of the actual protocol, only
that running the testcase should be considered unsuccessful if the
program exits with a failure code (in a POSIX system; in some other
environment, there may be other indications for failure of execution).

> Instead, DejaGnu expects an explicit "END" token from the unit test
> program to indicate that the program has reached its intended
> completion.  A warning is produced if this token is not observed;
> perhaps a future version of DejaGnu should insert an UNRESOLVED result
> like we currently do when a Tcl test script aborts?

I think inserting an UNRESOLVED result would be more appropriate.
Otherwise, there would be no formal failure result if a unit test
breaks due to, say, a segfault.

libgccjit uses a heavily patched version of host_execute ([1]) that
also handles tests run under Valgrind; perhaps some of the ideas from
that version can be incorporated into DejaGnu proper.

-- Marc

[1] https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89

>
>
> -- Jacob
>
>

Message #14 received at 79077 <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
Cc: 79077 <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Thu, 24 Jul 2025 21:13:53 -0500

[Message part 1 (text/plain, inline)]

On 7/24/25 00:57, Marc Nieper-Wißkirchen wrote:
> Hi Jacob,
>
> Thank you for your quick response!

You are welcome.

Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a 
thanks in the ChangeLog?

> Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer<jcb62281 <at> gmail.com>:
>> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
>>> Hi,
>>>
>>> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
>>> check the exit status of the executed test programs. A test program
>>> that simply aborts (e.g. using the C function of the same name) won't
>>> cause any testsuite failures. This seems brittle and like a
>>> misfeature.
>>>
>>> What is the supposed way to deal with this?
>> The host_execute procedure is intended for running unit test programs
>> that speak a special DejaGnu protocol and ignoring exit codes from unit
>> test programs is intended.  The DejaGnu unit testing protocol does not
>> depend on the exit code of the unit test program at all, because that
>> exit code might not be available in all environments.
> I am sorry if I wasn't clear enough in my previous email. I didn't
> mean that the exit code should be part of the actual protocol, only
> that running the testcase should be considered unsuccessful if the
> program exits with a failure code (in a POSIX system; in some other
> environment, there may be other indications for failure of execution).

But that would make the exit code part of the protocol.  DejaGnu 
generally supports running tests on "remote" target boards and a target 
connected over a serial line is unlikely to return an exit code.  The 
only way to be sure that a dependency on the exit code will not creep in 
is to ignore the exit code.

Admittedly, DejaGnu does not yet properly support running unit tests on 
remote targets.

>> Instead, DejaGnu expects an explicit "END" token from the unit test
>> program to indicate that the program has reached its intended
>> completion.  A warning is produced if this token is not observed;
>> perhaps a future version of DejaGnu should insert an UNRESOLVED result
>> like we currently do when a Tcl test script aborts?
> I think inserting an UNRESOLVED result would be more appropriate.
> Otherwise, there would be no formal failure result if a unit test
> breaks due to, say, a segfault.

The explicit "END" token is a relatively recent addition (added after 
the last release).  I was reluctant to outright require it (in case 
there are any testsuites out there with independent unit test protocol 
implementations) but I now agree that a warning from the test framework 
is not enough when a unit test bombs out early.

An initial solution has been pushed to Savannah on the PR79077 branch. 
DejaGnu can now be run directly from a Git checkout, you should be able 
to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make 
check" command line.

> libgccjit uses a heavily patched version of host_execute ([1]) that
> also handles tests run under Valgrind; perhaps some of the ideas from
> that version can be incorporated into DejaGnu proper.

That heavily patched version has problems that have since been fixed 
upstream.  I would encourage you to try getting rid of it.

You should be able to move the Valgrind support into a wrapper around 
host_execute, since host_execute has accepted arguments since 1.6.3; you 
can test precisely by matching [info body host_execute] against "The 
arguments are".  Note that the name of the executable needs to be 
absolute; the DejaGnu utility proc "which" will search PATH for its 
argument.

A first draft (untested) off the top of my head:

proc wrap_host_execute {args} {
    global env

    set run_under_valgrind [info exists env(RUN_UNDER_VALGRIND)]

    set exec_args $args
    set executable [lindex $args 0]
    set arguments [lrange $args 1 end]

    if { $run_under_valgrind } {
	set valgrind_logfile "${executable}.valgrind.txt"
	set valgrind_params [which "valgrind"]
	lappend valgrind_params "--leak-check=full"
	lappend valgrind_params "--log-file=${valgrind_logfile}"
	set exec_args [eval [list linsert $exec_args 0] $valgrind_params]
    }

    set reported_error [eval [list host_execute] $exec_args]

    if { $run_under_valgrind } {
	upvar 2 name name
	parse_valgrind_logfile $name $valgrind_logfile xfail
    }

    return $reported_error
}

You should probably instead use a VALGRIND global and have one of the 
initialization files default it to [which "valgrind"] if 
RUN_UNDER_VALGRIND is set in the environment.  I would recommend taking 
VALGRIND from the first of:

    (1)  value given on DejaGnu command line ([info exists VALGRIND])
    (2)  environment variable VALGRIND ($::env(VALGRIND) in Tcl)
    (3)  default by searching PATH ([which "valgrind"])

A first draft (untested) for that initialization code off the top of my 
head:

if { ![info exists VALGRIND] && [info exists ::env(RUN_UNDER_VALGRIND)] } {
    if { [info exists ::env(VALGRIND)] } {
	set VALGRIND $::env(VALGRIND)
    } else {
	set VALGRIND [which "valgrind"]
    }
}

The code does nothing if VALGRIND is already set because the framework 
will set it as a Tcl global if it is given on the command line.  A 
framework-provided value can also originate from multi-pass testing and 
vary between passes, perhaps to compare results from different Valgrind 
versions.

> [1]https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/testsuite/jit.dg/jit.exp;h=57b133b6d8c6ba7424d4a97bef1ba34264d469be;hb=HEAD#l89

-- Jacob

[Message part 2 (text/html, inline)]

Message #17 received at 79077 <at> debbugs.gnu.org (full text, mbox):

From: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
To: jcb62281 <at> gmail.com
Cc: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>,
 79077 <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Fri, 25 Jul 2025 13:30:33 +0200

Hi,

Am Fr., 25. Juli 2025 um 04:14 Uhr schrieb Jacob Bachmeyer <jcb62281 <at> gmail.com>:

[...]

> Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a thanks in the ChangeLog?

Yes, exactly.

> Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer <jcb62281 <at> gmail.com>:
>
> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
>
> Hi,
>
> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
> check the exit status of the executed test programs. A test program
> that simply aborts (e.g. using the C function of the same name) won't
> cause any testsuite failures. This seems brittle and like a
> misfeature.
>
> What is the supposed way to deal with this?
>
> The host_execute procedure is intended for running unit test programs
> that speak a special DejaGnu protocol and ignoring exit codes from unit
> test programs is intended.  The DejaGnu unit testing protocol does not
> depend on the exit code of the unit test program at all, because that
> exit code might not be available in all environments.
>
> I am sorry if I wasn't clear enough in my previous email. I didn't
> mean that the exit code should be part of the actual protocol, only
> that running the testcase should be considered unsuccessful if the
> program exits with a failure code (in a POSIX system; in some other
> environment, there may be other indications for failure of execution).
>
> But that would make the exit code part of the protocol.  DejaGnu generally supports running tests on "remote" target boards and a target connected over a serial line is unlikely to return an exit code.  The only way to be sure that a dependency on the exit code will not creep in is to ignore the exit code.

With the recent addition of the "END" token, this is probably moot;
without, how would have DejaGnu handled a test running remotely, say
over a serial line, where the connection drops early? I saw a non-zero
exit code as an analogue of such a dropped connection, which could be
expressed by an "UNRESOLVED" result.

>
> Admittedly, DejaGnu does not yet properly support running unit tests on remote targets.
>
> Instead, DejaGnu expects an explicit "END" token from the unit test
> program to indicate that the program has reached its intended
> completion.  A warning is produced if this token is not observed;
> perhaps a future version of DejaGnu should insert an UNRESOLVED result
> like we currently do when a Tcl test script aborts?
>
> I think inserting an UNRESOLVED result would be more appropriate.
> Otherwise, there would be no formal failure result if a unit test
> breaks due to, say, a segfault.
>
> The explicit "END" token is a relatively recent addition (added after the last release).  I was reluctant to outright require it (in case there are any testsuites out there with independent unit test protocol implementations) but I now agree that a warning from the test framework is not enough when a unit test bombs out early.
>
> An initial solution has been pushed to Savannah on the PR79077 branch. DejaGnu can now be run directly from a Git checkout, you should be able to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make check" command line.

That is great news. Thank you! Moreover, thank you very much for your
ideas about the Valgrind mapper; very helpful, indeed.

Best regards,

Marc

[...]

Message #22 received at 79077-done <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Marc Nieper-Wißkirchen <marc.nieper+gnu <at> gmail.com>
Cc: 79077-done <at> debbugs.gnu.org
Subject: Re: bug#79077: host_execute and non-zero exit status
Date: Fri, 25 Jul 2025 23:21:19 -0500

[Message part 1 (text/plain, inline)]

On 7/25/25 06:30, Marc Nieper-Wißkirchen wrote:
> Hi,
>
> Am Fr., 25. Juli 2025 um 04:14 Uhr schrieb Jacob Bachmeyer<jcb62281 <at> gmail.com>:
>
> [...]
>
>> Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer<jcb62281 <at> gmail.com>:
>>
>> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
>> [...]
>>
>> [...]  DejaGnu generally supports running tests on "remote" target boards and a target connected over a serial line is unlikely to return an exit code.  The only way to be sure that a dependency on the exit code will not creep in is to ignore the exit code.
> With the recent addition of the "END" token, this is probably moot;
> without, how would have DejaGnu handled a test running remotely, say
> over a serial line, where the connection drops early? I saw a non-zero
> exit code as an analogue of such a dropped connection, which could be
> expressed by an "UNRESOLVED" result.

Previously, DejaGnu gave no indication that anything was wrong if a unit 
test program failed to complete, although older versions used a "Totals" 
line as an implicit end marker.  This was changed when the unit testing 
protocol was documented to avoid possible misfires if a unit test 
happens to emit a line beginning with "Totals"; the new "END" marker 
uses a syntax that is explicitly documented as reserved for unit test 
messages to the framework.

>> [...]
>>
>> An initial solution has been pushed to Savannah on the PR79077 branch. DejaGnu can now be run directly from a Git checkout, you should be able to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make check" command line.
> That is great news. Thank you! Moreover, thank you very much for your
> ideas about the Valgrind mapper; very helpful, indeed.

You are welcome.

Since an improvement has been merged to Git master, this closes the bug 
report.


-- Jacob

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.