Subject: bug#65178: Shepherd stops responding during "guix system reconfigure"

From: "Timo Wilken" <[email-masked]>
Date: Thu, 14 Dec 2023 23:55:47 +0100

After a bit of searching, it looks like 67538, 67230 and 65178 may be the same
issue.

Attila Lendvai wrote:
> > > my suspicion is that it's due to some error coming from a start
> > > GEXP that somehow derails shepherd's event loop.
> >
> > iirc I once managed to get a debugger out when it happened and it's
> > stuck waiting in one of the epoll/select/alike calls,
>
> ...or one of the start/stop GEXP's calls something that (sometimes?) blocks
> indefinitely (which violates the API of shepherd).

Same symptoms here again.

For context: this time I was trying to deploy some OCI/Docker containers using
Guix' `oci-container-service-type', specifically a Shepherd service called
"conduit". My code is here:

https://cgit.twilken.net/dotfiles/log/?h=matrix-containers

(Specifically, commits bf94f7872a1df293bd904bbd2c1ef7229f4f98a8 and
c87dcdae79c6266ac3dac70af08fbef5eb21629b.)

This is with Guix commit 1b2505217cf222d98cc960b8510660976a01cfa1.

I first ran "guix system reconfigure -L . tw/system/lud.scm" with commit
bf94f7872a1df293bd904bbd2c1ef7229f4f98a8, which had a bug (an env var was
wrong, so the container failed to start). This worked as expected in that
Shepherd tried to start the service, which failed, so Shepherd disabled it.

Then, I fixed the env var and re-ran "guix system reconfigure -L .
tw/system/lud.scm" with commit c87dcdae79c6266ac3dac70af08fbef5eb21629b.
Shepherd loaded the new "conduit" service fine, as far as I can tell, but
didn't restart it because it was still disabled.

I then enabled and started the service manually. Enabling worked fine, but on
start, I got no terminal output from Shepherd, and it hung.

I still had an error in my setup (directory permissions were wrong), and I got
a message in /var/log/messages to that effect:

--8<---------------cut here---------------start------------->8---
Dec 14 21:33:50 localhost shepherd[1]: Service conduit is currently disabled. 
Dec 14 21:34:04 localhost shepherd[1]: Enabled service conduit. 
Dec 14 21:34:07 localhost shepherd[1]: Starting service user-homes... 
Dec 14 21:34:07 localhost shepherd[1]: Service user-homes has been started. 
Dec 14 21:34:07 localhost shepherd[1]: Service user-homes started. 
Dec 14 21:34:07 localhost shepherd[1]: Service user-homes running with value #t. 
Dec 14 21:34:07 localhost shepherd[1]: Starting service conduit... 
Dec 14 21:34:07 localhost shepherd[1]: Service conduit has been started. 
Dec 14 21:34:07 localhost shepherd[1]: Service conduit started. 
Dec 14 21:34:07 localhost shepherd[1]: Service conduit running with value 13226. 
Dec 14 21:34:07 localhost shepherd[1]: [docker] conduit: [...] "IO error: While open a file for appending: /var/lib/matrix-conduit/LOG: Permission denied"
--8<---------------cut here---------------end--------------->8---

...showing that Shepherd had at least tried to start the new container. The
container is not running, though (due to the error shown above), and nothing
with PID 13226 is running.

The "herd start conduit" command did not return, and ^C-ing it did not help.
Afterwards, every "herd" command also hung without any output.

Here are the last four lines of the output of "sudo strace -s1000 herd status"
on such a hung machine:

--8<---------------cut here---------------start------------->8---
connect(10, {sa_family=AF_UNIX, sun_path="/var/run/shepherd/socket"}, 26) = 0
getcwd("/home/timo", 100)               = 11
write(10, "(shepherd-command (version 0) (action status) (service root) (arguments ()) (directory \"/home/timo\"))", 101) = 101
read(10,
--8<---------------cut here---------------end--------------->8---

The "read(10, " call never completes.

At least in this case, Shepherd still seems to be processing inbound inet
connections, so I can open new SSH connections to the machine.

Attaching to PID 1 with strace shows it is stuck in "epoll_wait(13, "
(unsurprisingly, fd 13 points to "anon_inode:[eventpoll]"). Here's a backtrace
of all threads in "gdb -p 1":

--8<---------------cut here---------------start------------->8---
(gdb) info threads
  Id   Target Id                                     Frame 
* 1    Thread 0x7f786544c380 (LWP 1) "shepherd"      0x00007f7865552626 in epoll_wait ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  2    Thread 0x7f7864e16640 (LWP 186) "GC-marker-0" 0x00007f78654cf16a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  3    Thread 0x7f7864615640 (LWP 187) "GC-marker-1" 0x00007f78654cf16a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  4    Thread 0x7f7863e14640 (LWP 188) "GC-marker-2" 0x00007f78654cf16a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  5    Thread 0x7f78634c6640 (LWP 190) "shepherd"    0x00007f786554300c in read ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
(gdb) thread apply all bt

Thread 5 (Thread 0x7f78634c6640 (LWP 190) "shepherd"):
#0  0x00007f786554300c in read () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f7865a48cc7 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#2  0x00007f78659427d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f786594438c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f786594e83c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007f7865a65455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007f7865a4d570 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#7  0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#8  0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#9  0x00007f78659e5b3e in scm_call_with_unblocked_asyncs () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#10 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#11 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007f7865a6b0f3 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007f78659e7e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007f78659e95ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007f7865a90092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007f7865a6be1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007f78659ea396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007f7865a6b049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007f786594e7fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#22 0x00007f7865a64c5d in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#24 0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 4 (Thread 0x7f7863e14640 (LWP 188) "GC-marker-2"):
#0  0x00007f78654cf16a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f78654d17e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007f7865948740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f7865948897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 3 (Thread 0x7f7864615640 (LWP 187) "GC-marker-1"):
#0  0x00007f78654cf16a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f78654d17e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007f7865948740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f7865948897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 2 (Thread 0x7f7864e16640 (LWP 186) "GC-marker-0"):
#0  0x00007f78654cf16a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f78654d17e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007f7865948740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f7865948897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f78654d23aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007f7865552f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 1 (Thread 0x7f786544c380 (LWP 1) "shepherd"):
#0  0x00007f7865552626 in epoll_wait () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007f7862bb9335 in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#2  0x00007f78659427d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007f786594438c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007f786594e83c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007f7865a65455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007f7862bb96ce in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#7  0x00007f78606246c2 in ?? ()
#8  0x00007f78620ba628 in ?? ()
#9  0x00007f7860627610 in ?? ()
#10 0x00007f786520ad80 in ?? ()
#11 0x00007f7865a14edc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007f7865a71215 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007f78659e9977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007f7865a1dff9 in scm_primitive_load () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007f78659e9977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007f78659ef846 in scm_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007f7865a4e3e6 in scm_shell () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007f7865a008cc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#22 0x00007f78659e7e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007f7865a71390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#24 0x00007f7865a7edb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#25 0x00007f78659e95ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#26 0x00007f7865a90092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#27 0x00007f7865a6be1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#28 0x00007f78659ea396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#29 0x00007f7865a6b049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#30 0x00007f786594e7fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#31 0x00007f7865a653f8 in scm_with_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#32 0x00007f7865a098e5 in scm_boot_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#33 0x00000000004010f7 in ?? ()
#34 0x00007f78654761f7 in __libc_start_call_main () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#35 0x00007f78654762ac in __libc_start_main_impl () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#36 0x0000000000401171 in ?? ()
--8<---------------cut here---------------end--------------->8---

Unrelatedly, I also have another Shepherd on a different machine that became
stuck after I ran a bunch of "guix system reconfigure" commands. The
backtraces there, if it helps:

--8<---------------cut here---------------start------------->8---
(gdb) info threads 
  Id   Target Id                                     Frame 
* 1    Thread 0x7ffaceef2380 (LWP 1) "shepherd"      0x00007fface938626 in epoll_wait ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  2    Thread 0x7fface1aa640 (LWP 231) "GC-marker-0" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  3    Thread 0x7ffacd9a9640 (LWP 232) "GC-marker-1" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  4    Thread 0x7ffacd1a8640 (LWP 233) "GC-marker-2" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  5    Thread 0x7ffacc9a7640 (LWP 234) "GC-marker-3" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  6    Thread 0x7ffacc1a6640 (LWP 235) "GC-marker-4" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  7    Thread 0x7ffacb9a5640 (LWP 236) "GC-marker-5" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  8    Thread 0x7ffacb1a4640 (LWP 237) "GC-marker-6" 0x00007fface8b516a in __futex_abstimed_wait_common ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  9    Thread 0x7ffaca832640 (LWP 249) "shepherd"    0x00007fface92900c in read ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
  10   Thread 0x7ffac89ca640 (LWP 26693) "shepherd"  0x00007fface92900c in read ()
   from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
(gdb) thread apply all bt

Thread 10 (Thread 0x7ffac89ca640 (LWP 26693) "shepherd"):
#0  0x00007fface92900c in read () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007ffacedf0e57 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#2  0x00007ffaced3c7d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced3e38c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007ffaced4883c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007ffacee62455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007ffacedf903d in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#7  0x00007ffacede4e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#8  0x00007ffac6832022 in ?? ()
#9  0x00007fface4d97f0 in ?? ()
#10 0x00007ffac94766c0 in ?? ()
#11 0x00007fface5f4b40 in ?? ()
#12 0x00007ffacee11edc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007ffacee6e215 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007ffacede65ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007ffacee8d092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007ffacee68e1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007ffacede7396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007ffacee68049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007ffaced487fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#21 0x00007ffacee623f8 in scm_with_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#22 0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#23 0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 9 (Thread 0x7ffaca832640 (LWP 249) "shepherd"):
#0  0x00007fface92900c in read () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007ffacee45cc7 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#2  0x00007ffaced3c7d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced3e38c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007ffaced4883c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007ffacee62455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007ffacee4a570 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#7  0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#8  0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#9  0x00007ffacede2b3e in scm_call_with_unblocked_asyncs () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#10 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#11 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007ffacee680f3 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007ffacede4e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007ffacede65ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007ffacee8d092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007ffacee68e1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007ffacede7396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007ffacee68049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007ffaced487fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#22 0x00007ffacee61c5d in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#24 0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 8 (Thread 0x7ffacb1a4640 (LWP 237) "GC-marker-6"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 7 (Thread 0x7ffacb9a5640 (LWP 236) "GC-marker-5"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--c
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 6 (Thread 0x7ffacc1a6640 (LWP 235) "GC-marker-4"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 5 (Thread 0x7ffacc9a7640 (LWP 234) "GC-marker-3"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 4 (Thread 0x7ffacd1a8640 (LWP 233) "GC-marker-2"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 3 (Thread 0x7ffacd9a9640 (LWP 232) "GC-marker-1"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 2 (Thread 0x7fface1aa640 (LWP 231) "GC-marker-0"):
#0  0x00007fface8b516a in __futex_abstimed_wait_common () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007fface8b77e8 in pthread_cond_wait@@GLIBC_2.3.2 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#2  0x00007ffaced42740 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced42897 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007fface8b83aa in start_thread () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#5  0x00007fface938f7c in clone3 () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6

Thread 1 (Thread 0x7ffaceef2380 (LWP 1) "shepherd"):
#0  0x00007fface938626 in epoll_wait () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#1  0x00007ffac9efc335 in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#2  0x00007ffaced3c7d1 in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#3  0x00007ffaced3e38c in ?? () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#4  0x00007ffaced4883c in GC_do_blocking () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#5  0x00007ffacee62455 in scm_without_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#6  0x00007ffac9efc6ce in ?? () from /gnu/store/h4nsywbhn8b4qyh40fhykk3q40qkr3wd-guile-fibers-1.3.1/lib/guile/3.0/extensions/fibers-epoll.so
#7  0x00007ffac76416c2 in ?? ()
#8  0x00007ffac934f594 in ?? ()
#9  0x00007ffac9476d83 in ?? ()
#10 0x00007fface5f4d80 in ?? ()
#11 0x00007ffacee11edc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#12 0x00007ffacee6e652 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#13 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#14 0x00007ffacede6977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#15 0x00007ffacee1aff9 in scm_primitive_load () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#16 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#17 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#18 0x00007ffacede6977 in scm_primitive_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#19 0x00007ffacedec846 in scm_eval () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#20 0x00007ffacee4b3e6 in scm_shell () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#21 0x00007ffacedfd8cc in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#22 0x00007ffacede4e1a in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#23 0x00007ffacee6e390 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#24 0x00007ffacee7bdb5 in scm_call_n () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#25 0x00007ffacede65ca in scm_call_2 () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#26 0x00007ffacee8d092 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#27 0x00007ffacee68e1f in scm_c_catch () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#28 0x00007ffacede7396 in scm_c_with_continuation_barrier () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#29 0x00007ffacee68049 in ?? () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#30 0x00007ffaced487fa in GC_call_with_stack_base () from /gnu/store/k1ha4n9v8d7myiiszvl2ic7xnb56l219-libgc-8.2.2/lib/libgc.so.1
#31 0x00007ffacee623f8 in scm_with_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#32 0x00007ffacee068e5 in scm_boot_guile () from /gnu/store/n24l8hxn6nvb7lz7zjlyd7i05khrm0i4-guile-3.0.9/lib/libguile-3.0.so.1
#33 0x00000000004010f7 in ?? ()
#34 0x00007fface85c1f7 in __libc_start_call_main () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#35 0x00007fface85c2ac in __libc_start_main_impl () from /gnu/store/ln6hxqjvz6m9gdd9s97pivlqck7hzs99-glibc-2.35/lib/libc.so.6
#36 0x0000000000401171 in ?? ()
--8<---------------cut here---------------end--------------->8---