GNU bug report logs -
#56424
hydra.nixos coverage job times out
Previous Next
To reply to this bug, email your comments to 56424 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Wed, 06 Jul 2022 17:48:01 GMT)
Full text and
rfc822 format available.
Message #3 received at submit <at> debbugs.gnu.org (full text, mbox):
Package: emacs
Version: 29.0.50
Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk
The hydra.nixos "coverage" job, which is the one that runs "make check",
had been timing out for a month. If this has not been flagged before
now, it seems like a sign that this automated build isn't useful any
more and should be stopped.
The way to do that is probably to open an issue at
https://github.com/NixOS/hydra/issues
There is also https://lists.gnu.org/mailman/listinfo/hydra-users ,
but I'm not sure it is used/monitored any more.
The start of the timeout cannot be pin-pointed, because the build was
broken for several days.
It last worked consistently around
https://hydra.nixos.org/eval/1764830
Then the build was broken for several days from:
https://hydra.nixos.org/eval/1764836
The build started working again in:
https://hydra.nixos.org/eval/1765592
but coverage was already timing out.
It has worked once or twice since then, but mostly timed out.
See https://hydra.nixos.org/jobset/gnu/emacs-trunk/evals?page=30
(at time of writing)
No-one has login access to this system. If you want to investigate this,
your options are:
1) Install Nix locally and try the "Reproduce locally" button from the
"Actions" menu on a job web page.
2) Make the check process more verbose, so you can see where it hangs.
The enevironment variable EMACS_HYDRA_CI is set in these builds.
3) Modify the hydra build recipe. To do this, get access to
https://savannah.nongnu.org/p/hydra-recipes in the usual way, and modify
the file emacs/release.nix.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Thu, 07 Jul 2022 09:05:01 GMT)
Full text and
rfc822 format available.
Message #6 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Glenn Morris <rgm <at> gnu.org> writes:
> Ref: https://hydra.nixos.org/jobset/gnu/emacs-trunk
>
> The hydra.nixos "coverage" job, which is the one that runs "make check",
> had been timing out for a month. If this has not been flagged before
> now, it seems like a sign that this automated build isn't useful any
> more and should be stopped.
Doesn't seem to be timing out now, but it's failing.
https://hydra.nixos.org/build/182893714/nixlog/1
lisp/dnd-tests.log:
FAILED dnd-tests-begin-drag-files "Method `mock' should specify both encoding and decoding command or an scp program"
FAILED dnd-tests-begin-file-drag "Method `mock' should specify both encoding and decoding command or an scp program"
I've added Po to the CCs.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Thu, 07 Jul 2022 10:08:01 GMT)
Full text and
rfc822 format available.
Message #9 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Lars Ingebrigtsen <larsi <at> gnus.org> writes:
> Doesn't seem to be timing out now, but it's failing.
>
> https://hydra.nixos.org/build/182893714/nixlog/1
>
> lisp/dnd-tests.log:
> FAILED dnd-tests-begin-drag-files "Method `mock' should specify both encoding and decoding command or an scp program"
> FAILED dnd-tests-begin-file-drag "Method `mock' should specify both encoding and decoding command or an scp program"
>
> I've added Po to the CCs.
Does the build system have scp installed?
Also, most of the remote file test code involved here wasn't written by
me. I think it was Michael Albinus.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Thu, 07 Jul 2022 17:35:02 GMT)
Full text and
rfc822 format available.
Message #12 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Po Lu <luangruo <at> yahoo.com> writes:
>> Doesn't seem to be timing out now, but it's failing.
>>
>> https://hydra.nixos.org/build/182893714/nixlog/1
>>
>> lisp/dnd-tests.log:
>> FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>> FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>> both encoding and decoding command or an scp program"
>>
>> I've added Po to the CCs.
>
> Does the build system have scp installed?
>
> Also, most of the remote file test code involved here wasn't written by
> me. I think it was Michael Albinus.
Right; adding Michael to the CCs.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Sun, 10 Jul 2022 02:03:01 GMT)
Full text and
rfc822 format available.
Message #15 received at 56424 <at> debbugs.gnu.org (full text, mbox):
You happened to look at one job that for some reason did not time out.
Browsing the past month of jobs shows timeout after timeout.
With increased logging, and a serial build, we can see that it times out
in process-tests:
https://hydra.nixos.org/build/183466954
Again; if no-one is looking at this CI system any more (which is fine),
then it would be better to stop wasting the electricity.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Sun, 10 Jul 2022 10:45:02 GMT)
Full text and
rfc822 format available.
Message #18 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Lars Ingebrigtsen <larsi <at> gnus.org> writes:
Hi,
>>> Doesn't seem to be timing out now, but it's failing.
>>>
>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>
>>> lisp/dnd-tests.log:
>>> FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>> FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>> both encoding and decoding command or an scp program"
>>>
>>> I've added Po to the CCs.
>>
>> Does the build system have scp installed?
>>
>> Also, most of the remote file test code involved here wasn't written by
>> me. I think it was Michael Albinus.
>
> Right; adding Michael to the CCs.
Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
for hydra; let's see what happens.
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Sun, 10 Jul 2022 12:40:01 GMT)
Full text and
rfc822 format available.
Message #21 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Glenn Morris <rgm <at> gnu.org> writes:
Hi Glenn,
> With increased logging, and a serial build, we can see that it times out
> in process-tests:
> https://hydra.nixos.org/build/183466954
The incrimed test is process-tests/fd-setsize-no-crash/make-process.
From EMBA, I know that there are sometimes problems with this test
package, that's why we have suppressed
process-tests/multiple-threads-waiting on EMBA, for example. Maybe
something similar shall be done for HYDRA?
I've CC'ed Philipp Stephani, the author of this test package, for
further analysis.
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Sun, 10 Jul 2022 13:01:02 GMT)
Full text and
rfc822 format available.
Message #24 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Glenn Morris <rgm <at> gnu.org> writes:
> Again; if no-one is looking at this CI system any more (which is fine),
> then it would be better to stop wasting the electricity.
I look at both Nixos and EMBA regularly, but if I see that it's all red,
I assume that the CI system has a problem (which is correct 93% of the
time, because those systems seem both odd and catastrophically
underpowered) and wait for somebody to reboot them.
I'd love for somebody to finally set up a real CI system with reasonable
power (so that a build takes a few minutes instead of a few hours) and a
normal OS.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Sun, 10 Jul 2022 20:10:02 GMT)
Full text and
rfc822 format available.
Message #27 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Michael Albinus <michael.albinus <at> gmx.de> writes:
Hi,
>>>> Doesn't seem to be timing out now, but it's failing.
>>>>
>>>> https://hydra.nixos.org/build/182893714/nixlog/1
>>>>
>>>> lisp/dnd-tests.log:
>>>> FAILED dnd-tests-begin-drag-files "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>> FAILED dnd-tests-begin-file-drag "Method `mock' should specify
>>>> both encoding and decoding command or an scp program"
>>>>
>>>> I've added Po to the CCs.
>>>
>>> Does the build system have scp installed?
>>>
>>> Also, most of the remote file test code involved here wasn't written by
>>> me. I think it was Michael Albinus.
>>
>> Right; adding Michael to the CCs.
>
> Thanks for the heads-up. I've instrumented dnd-tests-begin-drag-files
> for hydra; let's see what happens.
The dnd-tests on hydra shall be fixed now.
Since this bug report is about process-tests, we shall keep it open.
Best regards, Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Tue, 19 Jul 2022 12:53:02 GMT)
Full text and
rfc822 format available.
Message #30 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Am So., 10. Juli 2022 um 14:40 Uhr schrieb Michael Albinus
<michael.albinus <at> gmx.de>:
>
> Glenn Morris <rgm <at> gnu.org> writes:
>
> Hi Glenn,
>
> > With increased logging, and a serial build, we can see that it times out
> > in process-tests:
> > https://hydra.nixos.org/build/183466954
>
> The incrimed test is process-tests/fd-setsize-no-crash/make-process.
> From EMBA, I know that there are sometimes problems with this test
> package, that's why we have suppressed
> process-tests/multiple-threads-waiting on EMBA, for example. Maybe
> something similar shall be done for HYDRA?
>
> I've CC'ed Philipp Stephani, the author of this test package, for
> further analysis.
My 2 cents: These timeouts/failures indicate real bugs in Emacs. They
are incredibly hard to debug due to the enormous complexity in Emacs's
asynchronous process handling code, but just skipping them won't make
the bugs go away. Somebody should at least try to dig into these
issues. But probably that requires something like the CI system
creating a snapshot of the failed tests as a Docker container that one
could SSH into, because the failures tend to be impossible to
reproduce locally.
Information forwarded
to
bug-gnu-emacs <at> gnu.org:
bug#56424; Package
emacs.
(Tue, 11 Oct 2022 01:04:02 GMT)
Full text and
rfc822 format available.
Message #33 received at 56424 <at> debbugs.gnu.org (full text, mbox):
Everything on Nixos seems to be failing now. For instance:
https://hydra.nixos.org/build/193879284
I'm not sure what the actual failure there is at all, but there's a
bunch of warnings like:
Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.32), passed through in regex; marked by <-- HERE in m/^\s+@([[:alnum:]][[:alnum:]\-]*)({ <-- HERE })?\s*/ at /nix/store/xbxabhr44naykgf768533sijfcamkyr9-texinfo-6.5/share/texinfo/Texinfo/Parser.pm line 5481.
The actual log ends with:
GEN ../../lisp/international/cp51932.el
GEN ../../lisp/international/eucjp-ms.el
GEN ../../etc/charsets/GB180304.map
GEN ../../etc/charsets/JISX2131.map
GEN charsets.stamp
make[3]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/admin/charsets'
make[2]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make[1]: *** [Makefile:537: src] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
make[1]: Entering directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source'
***
*** "make all" failed with exit status 2.
Which isn't very helpful.
Hm... Oh, the actual error is earlier:
mv: cannot stat 'lisp.mk.tmp': No such file or directory
GEN globals.h
GEN buildobj.h
GEN globals.h
GEN buildobj.h
cmp: globals.tmp: No such file or directory
mv: cannot stat 'globals.tmp': No such file or directory
cmp: globals.tmp: No such file or directory
make[1]: *** [Makefile:671: gl-stamp] Error 2
make[1]: Leaving directory '/build/bc1m3dggd3vw7606h3asba5nsbx3d3qz-source/src'
make: *** [Makefile:537: src] Error 2
make: *** Waiting for unfinished jobs....
Either something is very wrong on that build machine, or there's
something very wrong in our build.
This bug report was last modified 2 years and 248 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.