Subject: crypto with gnutls aka nettle (libhogweed) and scm_realloc

From: Linas Vepstas <[email-masked]>
Date: Sat, 2 Nov 2019 20:10:50 -0500

I've got an app that links gnutls (for crypto code) which links nettle
(libhogweed) with is a GMP-using crypto library which seems like it wanted
to call plain-old realloc, and ended up calling scm_realloc instead.  Note
that nettle does NOT use guile, so there's no plausible way that I know of
to end up in guile code.  This only seems to happen when nettle is used
from multiple threads (so is maybe a nettle bug??) but the stack trace is
so bizarre, I thought I'd report it here.

It would seem that someone, somewhere, is doing some low-level thunking or
trampolining of realloc().  First, the crazy stack trace:

It's currently highly reproducible and exact:
(gdb) r
Starting program:
/home/linas/src/novamente/src/atomspace-dht/build/tests/persist/dht/MultiUserUTest

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Running cxxtest tests (2 tests)Start creating 6 user sessions
[2019-11-03 00:46:03:350] [DEBUG] BEGIN TEST: test_multiuser
Collecting from unknown thread

Thread 13 "MultiUserUTest" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe0ff9700 (LWP 3844)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff6834535 in __GI_abort () at abort.c:79
#2  0x00007ffff5c80ded in GC_push_all_stacks () at pthread_stop_world.c:585
#3  0x00007ffff5c777df in GC_mark_some (
    cold_gc_frame=0x7fffe0ff59d0 "\274\327\354\365\377\177") at mark.c:322
#4  0x00007ffff5c6d15d in GC_stopped_mark (
    stop_func=stop_func@entry=0x7ffff5c6cbf0 <GC_never_stop_func>)
    at alloc.c:698
#5  0x00007ffff5c6dc69 in GC_try_to_collect_inner (
    stop_func=0x7ffff5c6cbf0 <GC_never_stop_func>) at alloc.c:486
#6  0x00007ffff5c6deea in GC_try_to_collect_general (
    stop_func=stop_func@entry=0x0, force_unmap=force_unmap@entry=0)
    at alloc.c:1065
#7  0x00007ffff5c6dfbd in GC_gcollect () at alloc.c:1089
#8  0x00007ffff6df3e5e in scm_gc_register_allocation (size=size@entry=136)
    at ../../libguile/gc.c:596
#9  0x00007ffff6df3554 in do_realloc (new_size=136, from=0x0)
    at ../../libguile/gc-malloc.c:70
#10 scm_realloc (mem=0x0, size=136) at ../../libguile/gc-malloc.c:117
#11 0x00007ffff630431f in _nettle_gmp_alloc ()
   from /usr/lib/x86_64-linux-gnu/libhogweed.so.4
#12 0x00007ffff62fc968 in nettle_mpz_random_size ()
   from /usr/lib/x86_64-linux-gnu/libhogweed.so.4
#13 0x00007ffff62fc9f4 in nettle_mpz_random ()
   from /usr/lib/x86_64-linux-gnu/libhogweed.so.4
#14 0x00007ffff62fcd63 in _nettle_generate_pocklington_prime ()
   from /usr/lib/x86_64-linux-gnu/libhogweed.so.4
#15 0x00007ffff62fd2ce in nettle_random_prime ()
   from /usr/lib/x86_64-linux-gnu/libhogweed.so.4
#16 0x00007ffff6300b53 in nettle_rsa_generate_keypair ()
   from /usr/lib/x86_64-linux-gnu/libhogweed.so.4
#17 0x00007ffff7e0729e in ?? () from
/usr/lib/x86_64-linux-gnu/libgnutls.so.30
#18 0x00007ffff7da8f07 in gnutls_x509_privkey_generate2 ()
   from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#19 0x00007ffff7f16990 in dht::crypto::PrivateKey::generate(unsigned int) ()
   from
/home/linas/src/novamente/src/atomspace-dht/build/opencog/persist/dht/libpersist-dht.so

Next, verify that nettle does not use scm:

$ nm /usr/lib/x86_64-linux-gnu/libhogweed.a |grep scm
(nothing printed)
$ nm /usr/lib/x86_64-linux-gnu/libhogweed.a |grep GC
(nothing printed)
$ nm /usr/lib/x86_64-linux-gnu/libhogweed.a |grep alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
                 U _nettle_gmp_alloc
00000000000005f0 T _nettle_gmp_alloc
00000000000004e0 T _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs
                 U _nettle_gmp_alloc_limbs

Debugging suggestions?
-- 
cassette tapes - analog TV - film cameras - you