GNU bug report logs -
#41824
Dejagnu's unknown proc aborts testsuite run when triggered in test-case
Previous Next
Full log
View this message in rfc822 format
Pedro Alves wrote:
> On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:
>
>
>>> The issue for me is that the "gdb Summary" tally did not
>>> mention the aborted testcase. I would like to see something
>>> like this instead:
>>>
>>> # of expected passes 208
>>> # of nasty OMG FIX! tcl errors 1
>>>
>>> :-)
>>>
>>> Your UNRESOLVED is of course better than the status quo.
>>>
>>>
>> UNRESOLVED is a step in the right direction, but I agree that more is needed. We are somewhat limited by POSIX, which does not define a separate result type for "test case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to require a testsuite to have a strictly-defined set of tests and that a run must produce results for exactly all of them, which requires information DejaGnu does not normally have about the running testsuite. So, while UNRESOLVED is needed, we could also maintain a separate count of UNRESOLVED-due-to-crash, or simply store away the error message/errorCode/errorInfo tuples and emit them a second time at the end of the run, also easy in Tcl 8.
>>
>
> I'm not sure how much is POSIX compliance important these days. I would hope that that
> would not limit DejaGnu's development and limit its potential to be more usable.
We already have extensions; POSIX only allowed
PASS/FAIL/UNTESTED/UNRESOLVED/UNSUPPORTED. [KX]{PASS,FAIL} are GNU
extensions.
> Even
> if POSIX compliance is important, I would think that that could be put behind
> a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like other
> GNU tools do:
>
> https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html
>
> Or conversely, we could have a --gnu mode, though I would say that a GNU tool
> should behave in GNU mode by default.
>
POSIX compliance in DejaGnu depended on the testsuite being run. Any
results generated by the framework itself need to be in the POSIX set,
though testsuites can use extensions, as GDB has done. Actually adding
more test result codes is dubious, as that will interact badly with XML
output and efficient database processing, both of which can reasonably
enumerate the nine result codes. ({,K,X}{PASS,FAIL},
UN{TEST,RESOLV,SUPPORT}ED)
> I really don't buy that adding different result types is an issue, as
> hinted earlier. We've actually done it in GDB, for a loooong while. We
> have distinct XFAIL vs KFAIL output messages show up in the .sum file (though
> those aren't distinguished in the summary file, doing so would be convenient,
> hence I see that as a GDB testsuite bug).
>
> And just very recently, we've added PATH and DUPLICATE output messages,
> and those do show up in the summary:
>
> # of expected passes 89593
> # of unexpected failures 547
> # of expected failures 146
> # of known failures 104
> # of untested testcases 20
> # of unresolved testcases 5
> # of unsupported tests 97
> # of paths in test names 1 <<<<<
> # of duplicate test names 370 <<<<<
>
> I would think that even these two would be useful to have in
> DejaGnu proper for all testsuites.
>
> v1/rationale - https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html
> v3/final version - https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html
>
I will look at the exact details later, but this sounds like it is
probably a monkeypatch that will stop working in some future version
when the procedures you are overriding disappear into an internal
namespace, so we need to be thinking about what would be needed to
provide extensible summary line sets, just as we need some way to extend
language support in default_target_compile without overriding the entire
procedure.
-- Jacob
This bug report was last modified 4 years and 313 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.