without depending on list-of-tests.mk, the dependencies of new tests are not extracted automatically, and so tests that depend on other tests can be run in the wrong order Indeed. For the record (as Reuben already knows :), I was seeing the same unpredictable behavior wrt the new vala tests (in #13002) not finding pkg-config. Thus I have pushed the following change to try to revert that part of the #11347 change. Fingers crossed. Thanks much. -k