Subject: bug#79077: host_execute and non-zero exit status

From: Marc Nieper-Wißkirchen <[email-masked]>
Date: Fri, 25 Jul 2025 13:30:33 +0200

Hi,

Am Fr., 25. Juli 2025 um 04:14 Uhr schrieb Jacob Bachmeyer <[email-masked]>:

[...]

> Is Nieper-Wisskirchen a proper ASCII transliteration of your name for a thanks in the ChangeLog?

Yes, exactly.

> Am Do., 24. Juli 2025 um 05:09 Uhr schrieb Jacob Bachmeyer <[email-masked]>:
>
> On 7/23/25 02:31, Marc Nieper-Wißkirchen wrote:
>
> Hi,
>
> The host_execute procedure in dejagnu.exp (see [1]) doesn't seem to
> check the exit status of the executed test programs. A test program
> that simply aborts (e.g. using the C function of the same name) won't
> cause any testsuite failures. This seems brittle and like a
> misfeature.
>
> What is the supposed way to deal with this?
>
> The host_execute procedure is intended for running unit test programs
> that speak a special DejaGnu protocol and ignoring exit codes from unit
> test programs is intended.  The DejaGnu unit testing protocol does not
> depend on the exit code of the unit test program at all, because that
> exit code might not be available in all environments.
>
> I am sorry if I wasn't clear enough in my previous email. I didn't
> mean that the exit code should be part of the actual protocol, only
> that running the testcase should be considered unsuccessful if the
> program exits with a failure code (in a POSIX system; in some other
> environment, there may be other indications for failure of execution).
>
> But that would make the exit code part of the protocol.  DejaGnu generally supports running tests on "remote" target boards and a target connected over a serial line is unlikely to return an exit code.  The only way to be sure that a dependency on the exit code will not creep in is to ignore the exit code.

With the recent addition of the "END" token, this is probably moot;
without, how would have DejaGnu handled a test running remotely, say
over a serial line, where the connection drops early? I saw a non-zero
exit code as an analogue of such a dropped connection, which could be
expressed by an "UNRESOLVED" result.

>
> Admittedly, DejaGnu does not yet properly support running unit tests on remote targets.
>
> Instead, DejaGnu expects an explicit "END" token from the unit test
> program to indicate that the program has reached its intended
> completion.  A warning is produced if this token is not observed;
> perhaps a future version of DejaGnu should insert an UNRESOLVED result
> like we currently do when a Tcl test script aborts?
>
> I think inserting an UNRESOLVED result would be more appropriate.
> Otherwise, there would be no formal failure result if a unit test
> breaks due to, say, a segfault.
>
> The explicit "END" token is a relatively recent addition (added after the last release).  I was reluctant to outright require it (in case there are any testsuites out there with independent unit test protocol implementations) but I now agree that a warning from the test framework is not enough when a unit test bombs out early.
>
> An initial solution has been pushed to Savannah on the PR79077 branch. DejaGnu can now be run directly from a Git checkout, you should be able to simply pass RUNTEST=/full/name/of/working/tree/runtest on the "make check" command line.

That is great news. Thank you! Moreover, thank you very much for your
ideas about the Valgrind mapper; very helpful, indeed.

Best regards,

Marc

[...]