GNU bug report logs - #52646
GC thread freeze

Previous Next

To reply to this bug, email your comments to 52646 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: bug-guile <at> gnu.org
Subject: GC thread freeze
Date: Sat, 18 Dec 2021 21:52:30 +0100

Hello,

I experiment a strange behaviour with this Guile 3.0.7 process:
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/scripts/remote-worker.scm.

The process is forking N processes that in turn start 4 threads. On
aarch64 machines specifically, some of those threads are freezing. Here
is what GDB is reporting:

--8<---------------cut here---------------start------------->8---
(gdb) attach 5660 ;frozen cuirass-remote-worker PID
(gdb) info thr
  Id   Target Id                                       Frame 
* 1    Thread 0xffffafd32e20 (LWP 5660) "yHg3r3fS"     0x0000ffffafb3fa80 in do_futex_wait.constprop () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libpthread.so.0
  2    Thread 0xffffa6c1c1d0 (LWP 5666) "ZMQbg/Reaper" 0x0000ffffaf7ec294 in epoll_pwait () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libc.so.6
  3    Thread 0xffffaf0071d0 (LWP 5667) "ZMQbg/IO/0"   0x0000ffffaf7ec294 in epoll_pwait () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libc.so.6
  4    Thread 0xffffa641b1d0 (LWP 5674) "yHg3r3fS"     0x0000ffffaf7b9d04 in clock_nanosleep@@GLIBC_2.17 () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libc.so.6
(gdb) bt
#0  0x0000ffffafb3fa80 in do_futex_wait.constprop () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libpthread.so.0
#1  0x0000ffffafb3fb78 in __new_sem_wait_slow.constprop.0 () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libpthread.so.0
#2  0x0000ffffafb80318 in GC_stop_world () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#3  0x0000ffffafb6c020 in GC_stopped_mark () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#4  0x0000ffffafb6c8dc in GC_try_to_collect_inner () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#5  0x0000ffffafb6d598 in GC_collect_or_expand () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#6  0x0000ffffafb73b4c in GC_alloc_large () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#7  0x0000ffffafb74038 in GC_generic_malloc () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#8  0x0000ffffafb74298 in GC_malloc_kind_global () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#9  0x0000ffffafc11fa8 in scm_make_bytevector () from /gnu/store/7g3nbnf2kf31jk696k0nyz9ck55b11a0-guile-3.0.7/lib/libguile-3.0.so.1
#10 0x0000ffffacacc418 in ?? ()
#11 0x0000ffffacc2ef2c in ?? ()
(gdb) thr 4
[Switching to thread 4 (Thread 0xffffa641b1d0 (LWP 5674))]
#0  0x0000ffffaf7b9d04 in clock_nanosleep@@GLIBC_2.17 () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libc.so.6
(gdb) bt
#0  0x0000ffffaf7b9d04 in clock_nanosleep@@GLIBC_2.17 () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libc.so.6
#1  0x0000ffffaf7bf55c in nanosleep () from /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/libc.so.6
#2  0x0000ffffafb7e844 in GC_lock () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#3  0x0000ffffafb7ecdc in GC_do_blocking_inner () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#4  0x0000ffffafb73998 in GC_with_callee_saves_pushed () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#5  0x0000ffffafb79654 in GC_do_blocking () from /gnu/store/jsda4njqwjp4kb60fwa7n4mlfi1aanpq-libgc-7.6.12/lib/libgc.so.1
#6  0x0000ffffafc96d94 in scm_without_guile () from /gnu/store/7g3nbnf2kf31jk696k0nyz9ck55b11a0-guile-3.0.7/lib/libguile-3.0.so.1
#7  0x0000ffffafc97050 in scm_std_select () from /gnu/store/7g3nbnf2kf31jk696k0nyz9ck55b11a0-guile-3.0.7/lib/libguile-3.0.so.1
#8  0x0000ffffafc97b5c in scm_std_sleep () from /gnu/store/7g3nbnf2kf31jk696k0nyz9ck55b11a0-guile-3.0.7/lib/libguile-3.0.so.1
#9  0x0000ffffafc75918 in scm_sleep () from /gnu/store/7g3nbnf2kf31jk696k0nyz9ck55b11a0-guile-3.0.7/lib/libguile-3.0.so.1
#10 0x0000ffffa6c50d94 in ?? ()
#11 0x0000ffffacc2ee0c in ?? ()
--8<---------------cut here---------------end--------------->8---

The threads 1 and 4 do no respond anymore and are stuck, thread 1 on a
futex wait and thread 4 on a sleep, both in the GC library. For what
it's worth, I do not experiment this behaviour on x86 machines.

I tried to come up with a smaller reproducer without success, but I'll
keep trying.

Thanks,

Mathieu

Message #8 received at 52646 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <othacehe <at> gnu.org>
Cc: 52646 <at> debbugs.gnu.org
Subject: Re: bug#52646: GC thread freeze
Date: Tue, 21 Dec 2021 11:16:36 +0100

Hello!

Mathieu Othacehe <othacehe <at> gnu.org> skribis:

> I experiment a strange behaviour with this Guile 3.0.7 process:
> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/scripts/remote-worker.scm.
>
> The process is forking N processes that in turn start 4 threads.

This is happening in this order, right?

POSIX leaves unspecified the behavior of a child process forked from a
multi-threaded process; there could be deadlocks, etc.  ‘primitive-fork’
prints a warning when called from a multi-threaded Guile process.

The solution is for multi-threaded Guile processes to not fork at all,
or to fork only via ‘open-pipe*’, ‘system*’, etc., which are “known
good” (they take care of post-fork handling in the child and call ‘exec’
before anything bad could happen.)

Thanks,
Ludo’.

Message #11 received at 52646 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 52646 <at> debbugs.gnu.org
Subject: Re: bug#52646: GC thread freeze
Date: Wed, 22 Dec 2021 10:15:06 +0100

Hey!

> This is happening in this order, right?

Right, plus I don't see the following warning:

--8<---------------cut here---------------start------------->8---
warning: call to primitive-fork while multiple threads are running;
         further behavior unspecified.  See "Processes" in the
         manual, for more information
--8<---------------cut here---------------end--------------->8---

so I guess the issue is elsewhere.

> POSIX leaves unspecified the behavior of a child process forked from a
> multi-threaded process; there could be deadlocks, etc.  ‘primitive-fork’
> prints a warning when called from a multi-threaded Guile process.
>
> The solution is for multi-threaded Guile processes to not fork at all,
> or to fork only via ‘open-pipe*’, ‘system*’, etc., which are “known
> good” (they take care of post-fork handling in the child and call ‘exec’
> before anything bad could happen.)

Thanks for clarifying that anyway.

Mathieu

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.