Thanks, Pip! Really appreciate you putting so much effort into this. See below for what you asked for.

George

It's a good idea to use "gcore" to generate a core file from gdb, just
in case the session is destroyed.

Done.

Can you check whether eln files were created just before the crash?
Over here, the relevant command would be

$ ls -Strl ~/.emacs.d/eln-cache/*/*

and looking at the last few lines to see whether any files have the
creation date in the right range (after you started using eshell but
before the crash happened).

Nothing in the directory ( /u/panagopo/.config/emacs/.local/cache/eln/30.1-1ed0c1e8 for me) that is on the same day of the crash. In fact, eshell isn't there at all, which probably makes sense as I am using the built-in version, so it was already natively-compiled.

  So we first should inspect the memory around 0x196922b0 to find out
whether it looks like a valid vector block, and whether the bad word was
in this block or in the string object marked just before.  I'd suggest
running

    x/32gx 0x196922b0

to look at the memory following the pointer, and

    x/32gx 0x19692200

to get some idea of whether it might be the middle of a vectorlike.

(gdb)   x/32gx 0x196922b0
0x196922b0: 0xc00000001f000005 0x0000000000000406
0x196922c0: 0x000015554efc8fac 0x000000000ffdf6e1
0x196922d0: 0x0000000000000016 0x000015554efc8f74
0x196922e0: 0xc00000001200a000 0x00001555386f8c00
0x196922f0: 0x0000000000000000 0x0000000074e43510
0x19692300: 0x0000000000000000 0x0000000000000000
0x19692310: 0x000000000000000b 0x00000000556ee565
0x19692320: 0x000000007a4401e0 0x0000000000000000
0x19692330: 0x000000005d1f2243 0xc000000003005000
0x19692340: 0x00000000134314f0 0x0000000000000000
0x19692350: 0x000000001965e118 0x00000000000026ba
0x19692360: 0x00000000000026ba 0x8000000000000004
0x19692370: 0x0000000000000f26 0x0000000000000c4e
0x19692380: 0x0000000000000c02 0x0000000000000d16
0x19692390: 0xc000000003005000 0x000015554ef1fbb0
0x196923a0: 0x00000000012d6144 0x0000000019e14120
(gdb)  x/32gx 0x19692200
0x19692200: 0x0000000000000000 0x0000000000019082
0x19692210: 0x0000000000019102 0x000000000000004e
0x19692220: 0x0000000000000000 0x0000000000000192
0x19692230: 0x0000000000000002 0x0000000000000000
0x19692240: 0x400000002300000d 0x0000000000000000
0x19692250: 0x000000000544a480 0x00000000019e1500
0x19692260: 0x0000000000000000 0x000000000000b640
0x19692270: 0x0000000000000000 0x0000000000019082
0x19692280: 0x0000000000019102 0x000000000000004e
0x19692290: 0x0000000000000000 0x0000000000000192
0x196922a0: 0x0000000000000002 0x0000000000000000
0x196922b0: 0xc00000001f000005 0x0000000000000406
0x196922c0: 0x000015554efc8fac 0x000000000ffdf6e1
0x196922d0: 0x0000000000000016 0x000015554efc8f74
0x196922e0: 0xc00000001200a000 0x00001555386f8c00
0x196922f0: 0x0000000000000000 0x0000000074e43510


You could try running

    p mem_find (0x196922b5)
    p *$

to see where the vector block (if it is one) was supposed to start.
(There's a slight chance this will crash). It may be worth it to print
the entire vector block, which should be doable by running

    x/512gx $.start

after that command.


 (gdb) p mem_find (0x196922b5)
[LWP 3311069 exited]
$7 = (struct mem_node *) 0x18f97010
(gdb) p *$
$8 = {left = 0x18f96fd0, right = 0x18f97050, parent = 0x18f96f90, start = 0x196915a0, end = 0x19692598, color = MEM_BLACK, type = MEM_TYPE_VECTOR_BLOCK}
(gdb) x/512gx $.start
0x196915a0: 0xc000000021000081 0x0000068000000003
0x196915b0: 0x000000000000158a 0x000000000000158e
0x196915c0: 0x0000000000001592 0x0000000000001596
0x196915d0: 0x000000000000159a 0x000000000000159e
0x196915e0: 0x00000000000015a2 0x00000000000015a6
0x196915f0: 0x00000000000015aa 0x00000000000015ae
0x19691600: 0x00000000000015b2 0x00000000000015b6
0x19691610: 0x00000000000015ba 0x00000000000015be
0x19691620: 0x00000000000015c2 0x00000000000015c6
0x19691630: 0x00000000000015ca 0x00000000000015ce
0x19691640: 0x00000000000015d2 0x00000000000015d6
0x19691650: 0x00000000000015da 0x00000000000015de
0x19691660: 0x00000000000015e2 0x00000000000015e6
0x19691670: 0x00000000000015ea 0x00000000000015ee
0x19691680: 0x00000000000015f2 0x00000000000015f6
0x19691690: 0x00000000000015fa 0x00000000000015fe
0x196916a0: 0x0000000000001602 0x0000000000001606
0x196916b0: 0x000000000000160a 0x000000000000160e
0x196916c0: 0x0000000000001612 0x0000000000001616
0x196916d0: 0x000000000000161a 0x000000000000161e
0x196916e0: 0x0000000000001622 0x0000000000001626
0x196916f0: 0x000000000000162a 0x000000000000162e
0x19691700: 0x0000000000001632 0x0000000000001636
0x19691710: 0x000000000000163a 0x000000000000163e
0x19691720: 0x0000000000001642 0x0000000000001646
0x19691730: 0x000000000000164a 0x000000000000164e
0x19691740: 0x0000000000001652 0x0000000000001656
0x19691750: 0x000000000000165a 0x000000000000165e
0x19691760: 0x0000000000001662 0x0000000000001666
0x19691770: 0x000000000000166a 0x000000000000166e
0x19691780: 0x0000000000001672 0x0000000000001676
0x19691790: 0x000000000000167a 0x000000000000167e
0x196917a0: 0x0000000000001682 0x0000000000001686
0x196917b0: 0x000000000000168a 0x000000000000168e
0x196917c0: 0x0000000000001692 0x0000000000001696
0x196917d0: 0x000000000000169a 0x000000000000169e
0x196917e0: 0x00000000000016a2 0x00000000000016a6
0x196917f0: 0x00000000000016aa 0x00000000000016ae
0x19691800: 0x00000000000016b2 0x00000000000016b6
0x19691810: 0x00000000000016ba 0x00000000000016be
0x19691820: 0x00000000000016c2 0x00000000000016c6
0x19691830: 0x00000000000016ca 0x00000000000016ce
0x19691840: 0x00000000000016d2 0x00000000000016d6
0x19691850: 0x00000000000016da 0x00000000000016de
0x19691860: 0x00000000000016e2 0x00000000000016e6
0x19691870: 0x00000000000016ea 0x00000000000016ee
0x19691880: 0x00000000000016f2 0x00000000000016f6
0x19691890: 0x00000000000016fa 0x00000000000016fe
0x196918a0: 0x0000000000001702 0x0000000000001706
0x196918b0: 0x000000000000170a 0x000000000000170e
0x196918c0: 0x0000000000001712 0x0000000000001716
0x196918d0: 0x000000000000171a 0x000000000000171e
0x196918e0: 0x0000000000001722 0x0000000000001726
--Type <RET> for more, q to quit, c to continue without paging--c
0x196918f0: 0x000000000000172a 0x000000000000172e
0x19691900: 0x0000000000001732 0x0000000000001736
0x19691910: 0x000000000000173a 0x000000000000173e
0x19691920: 0x0000000000001742 0x0000000000001746
0x19691930: 0x000000000000174a 0x000000000000174e
0x19691940: 0x0000000000001752 0x0000000000001756
0x19691950: 0x000000000000175a 0x000000000000175e
0x19691960: 0x0000000000001762 0x0000000000001766
0x19691970: 0x000000000000176a 0x000000000000176e
0x19691980: 0x0000000000001772 0x0000000000001776
0x19691990: 0x000000000000177a 0x000000000000177e
0x196919a0: 0x0000000000001782 0x0000000000001786
0x196919b0: 0xc000000021000081 0x0000070000000003
0x196919c0: 0x000000000000178a 0x000000000000178e
0x196919d0: 0x0000000000001792 0x0000000000001796
0x196919e0: 0x000000000000179a 0x000000000000179e
0x196919f0: 0x00000000000017a2 0x00000000000017a6
0x19691a00: 0x00000000000017aa 0x00000000000017ae
0x19691a10: 0x00000000000017b2 0x00000000000017b6
0x19691a20: 0x00000000000017ba 0x00000000000017be
0x19691a30: 0x00000000000017c2 0x00000000000017c6
0x19691a40: 0x00000000000017ca 0x00000000000017ce
0x19691a50: 0x00000000000017d2 0x00000000000017d6
0x19691a60: 0x00000000000017da 0x00000000000017de
0x19691a70: 0x00000000000017e2 0x00000000000017e6
0x19691a80: 0x00000000000017ea 0x00000000000017ee
0x19691a90: 0x00000000000017f2 0x00000000000017f6
0x19691aa0: 0x00000000000017fa 0x00000000000017fe
0x19691ab0: 0x0000000000001802 0x0000000000001806
0x19691ac0: 0x000000000000180a 0x000000000000180e
0x19691ad0: 0x0000000000001812 0x0000000000001816
0x19691ae0: 0x000000000000181a 0x000000000000181e
0x19691af0: 0x0000000000001822 0x0000000000001826
0x19691b00: 0x000000000000182a 0x000000000000182e
0x19691b10: 0x0000000000001832 0x0000000000001836
0x19691b20: 0x000000000000183a 0x000000000000183e
0x19691b30: 0x0000000000001842 0x0000000000001846
0x19691b40: 0x000000000000184a 0x000000000000184e
0x19691b50: 0x0000000000001852 0x0000000000001856
0x19691b60: 0x000000000000185a 0x000000000000185e
0x19691b70: 0x0000000000001862 0x0000000000001866
0x19691b80: 0x000000000000186a 0x000000000000186e
0x19691b90: 0x0000000000001872 0x0000000000001876
0x19691ba0: 0x000000000000187a 0x000000000000187e
0x19691bb0: 0x0000000000001882 0x0000000000001886
0x19691bc0: 0x000000000000188a 0x000000000000188e
0x19691bd0: 0x0000000000001892 0x0000000000001896
0x19691be0: 0x000000000000189a 0x000000000000189e
0x19691bf0: 0x00000000000018a2 0x00000000000018a6
0x19691c00: 0x00000000000018aa 0x00000000000018ae
0x19691c10: 0x00000000000018b2 0x00000000000018b6
0x19691c20: 0x00000000000018ba 0x00000000000018be
0x19691c30: 0x00000000000018c2 0x00000000000018c6
0x19691c40: 0x00000000000018ca 0x00000000000018ce
0x19691c50: 0x00000000000018d2 0x00000000000018d6
0x19691c60: 0x00000000000018da 0x00000000000018de
0x19691c70: 0x00000000000018e2 0x00000000000018e6
0x19691c80: 0x00000000000018ea 0x00000000000018ee
0x19691c90: 0x00000000000018f2 0x00000000000018f6
0x19691ca0: 0x00000000000018fa 0x00000000000018fe
0x19691cb0: 0x0000000000001902 0x0000000000001906
0x19691cc0: 0x000000000000190a 0x000000000000190e
0x19691cd0: 0x0000000000001912 0x0000000000001916
0x19691ce0: 0x000000000000191a 0x000000000000191e
0x19691cf0: 0x0000000000001922 0x0000000000001926
0x19691d00: 0x000000000000192a 0x000000000000192e
0x19691d10: 0x0000000000001932 0x0000000000001936
0x19691d20: 0x000000000000193a 0x000000000000193e
0x19691d30: 0x0000000000001942 0x0000000000001946
0x19691d40: 0x000000000000194a 0x000000000000194e
0x19691d50: 0x0000000000001952 0x0000000000001956
0x19691d60: 0x000000000000195a 0x000000000000195e
0x19691d70: 0x0000000000001962 0x0000000000001966
0x19691d80: 0x000000000000196a 0x000000000000196e
0x19691d90: 0x0000000000001972 0x0000000000001976
0x19691da0: 0x000000000000197a 0x000000000000197e
0x19691db0: 0x0000000000001982 0x0000000000001986
0x19691dc0: 0xc000000021000081 0x0000078000000003
0x19691dd0: 0x000000000000198a 0x000000000000198e
0x19691de0: 0x0000000000001992 0x0000000000001996
0x19691df0: 0x000000000000199a 0x000000000000199e
0x19691e00: 0x00000000000019a2 0x00000000000019a6
0x19691e10: 0x00000000000019aa 0x00000000000019ae
0x19691e20: 0x00000000000019b2 0x00000000000019b6
0x19691e30: 0x00000000000019ba 0x00000000000019be
0x19691e40: 0x00000000000019c2 0x00000000000019c6
0x19691e50: 0x00000000000019ca 0x00000000000019ce
0x19691e60: 0x00000000000019d2 0x00000000000019d6
0x19691e70: 0x00000000000019da 0x00000000000019de
0x19691e80: 0x00000000000019e2 0x00000000000019e6
0x19691e90: 0x00000000000019ea 0x00000000000019ee
0x19691ea0: 0x00000000000019f2 0x00000000000019f6
0x19691eb0: 0x00000000000019fa 0x00000000000019fe
0x19691ec0: 0x0000000000001a02 0x0000000000001a06
0x19691ed0: 0x0000000000001a0a 0x0000000000001a0e
0x19691ee0: 0x0000000000001a12 0x0000000000001a16
0x19691ef0: 0x0000000000001a1a 0x0000000000001a1e
0x19691f00: 0x0000000000001a22 0x0000000000001a26
0x19691f10: 0x0000000000001a2a 0x0000000000001a2e
0x19691f20: 0x0000000000001a32 0x0000000000001a36
0x19691f30: 0x0000000000001a3a 0x0000000000001a3e
0x19691f40: 0x0000000000001a42 0x0000000000001a46
0x19691f50: 0x0000000000001a4a 0x0000000000001a4e
0x19691f60: 0x0000000000001a52 0x0000000000001a56
0x19691f70: 0x0000000000001a5a 0x0000000000001a5e
0x19691f80: 0x0000000000001a62 0x0000000000001a66
0x19691f90: 0x0000000000001a6a 0x0000000000001a6e
0x19691fa0: 0x0000000000001a72 0x0000000000001a76
0x19691fb0: 0x0000000000001a7a 0x0000000000001a7e
0x19691fc0: 0x0000000000001a82 0x0000000000001a86
0x19691fd0: 0x0000000000001a8a 0x0000000000001a8e
0x19691fe0: 0x0000000000001a92 0x0000000000001a96
0x19691ff0: 0x0000000000001a9a 0x0000000000001a9e
0x19692000: 0x0000000000001aa2 0x0000000000001aa6
0x19692010: 0x0000000000001aaa 0x0000000000001aae
0x19692020: 0x0000000000001ab2 0x0000000000001ab6
0x19692030: 0x0000000000001aba 0x0000000000001abe
0x19692040: 0x0000000000001ac2 0x0000000000001ac6
0x19692050: 0x0000000000001aca 0x0000000000001ace
0x19692060: 0x0000000000001ad2 0x0000000000001ad6
0x19692070: 0x0000000000001ada 0x0000000000001ade
0x19692080: 0x0000000000001ae2 0x0000000000001ae6
0x19692090: 0x0000000000001aea 0x0000000000001aee
0x196920a0: 0x0000000000001af2 0x0000000000001af6
0x196920b0: 0x0000000000001afa 0x0000000000001afe
0x196920c0: 0x0000000000001b02 0x0000000000001b06
0x196920d0: 0x0000000000001b0a 0x0000000000001b0e
0x196920e0: 0x0000000000001b12 0x0000000000001b16
0x196920f0: 0x0000000000001b1a 0x0000000000001b1e
0x19692100: 0x0000000000001b22 0x0000000000001b26
0x19692110: 0x0000000000001b2a 0x0000000000001b2e
0x19692120: 0x0000000000001b32 0x0000000000001b36
0x19692130: 0x0000000000001b3a 0x0000000000001b3e
0x19692140: 0x0000000000001b42 0x0000000000001b46
0x19692150: 0x0000000000001b4a 0x0000000000001b4e
0x19692160: 0x0000000000001b52 0x0000000000001b56
0x19692170: 0x0000000000001b5a 0x0000000000001b5e
0x19692180: 0x0000000000001b62 0x0000000000001b66
0x19692190: 0x0000000000001b6a 0x0000000000001b6e
0x196921a0: 0x0000000000001b72 0x0000000000001b76
0x196921b0: 0x0000000000001b7a 0x0000000000001b7e
0x196921c0: 0x0000000000001b82 0x0000000000001b86
0x196921d0: 0x400000002300000d 0x0000000000000000
0x196921e0: 0x000000000544a480 0x00000000019e1500
0x196921f0: 0x0000000000000000 0x000000000000b640
0x19692200: 0x0000000000000000 0x0000000000019082
0x19692210: 0x0000000000019102 0x000000000000004e
0x19692220: 0x0000000000000000 0x0000000000000192
0x19692230: 0x0000000000000002 0x0000000000000000
0x19692240: 0x400000002300000d 0x0000000000000000
0x19692250: 0x000000000544a480 0x00000000019e1500
0x19692260: 0x0000000000000000 0x000000000000b640
0x19692270: 0x0000000000000000 0x0000000000019082
0x19692280: 0x0000000000019102 0x000000000000004e
0x19692290: 0x0000000000000000 0x0000000000000192
0x196922a0: 0x0000000000000002 0x0000000000000000
0x196922b0: 0xc00000001f000005 0x0000000000000406
0x196922c0: 0x000015554efc8fac 0x000000000ffdf6e1
0x196922d0: 0x0000000000000016 0x000015554efc8f74
0x196922e0: 0xc00000001200a000 0x00001555386f8c00
0x196922f0: 0x0000000000000000 0x0000000074e43510
0x19692300: 0x0000000000000000 0x0000000000000000
0x19692310: 0x000000000000000b 0x00000000556ee565
0x19692320: 0x000000007a4401e0 0x0000000000000000
0x19692330: 0x000000005d1f2243 0xc000000003005000
0x19692340: 0x00000000134314f0 0x0000000000000000
0x19692350: 0x000000001965e118 0x00000000000026ba
0x19692360: 0x00000000000026ba 0x8000000000000004
0x19692370: 0x0000000000000f26 0x0000000000000c4e
0x19692380: 0x0000000000000c02 0x0000000000000d16
0x19692390: 0xc000000003005000 0x000015554ef1fbb0
0x196923a0: 0x00000000012d6144 0x0000000019e14120
0x196923b0: 0x0000000000000094 0x0000000000000098
0x196923c0: 0x8000000000000001 0x00000000000018ea
0x196923d0: 0xc00000001a003007 0x000000001cfbfe44
0x196923e0: 0x000000001cf21683 0x000000001969242d
0x196923f0: 0x0000000019692475 0x00000000444a49c5
0x19692400: 0x000000001cfc4b65 0x000000001969c275
0x19692410: 0x000015552436e9e0 0x0000000000000000
0x19692420: 0x000000001cfc02a0 0xc00000000e008000
0x19692430: 0x000000001cfd6320 0x000000001cfd6190
0x19692440: 0x000000001cfd5b80 0x00000000006e9960
0x19692450: 0x000000001cfd59f0 0x0000001c0000001c
0x19692460: 0x0000400700000060 0x0000000000000000
0x19692470: 0xc00000000e008000 0x000000001cfd6cd0
0x19692480: 0x000000001cfd6b40 0x000000001cfd6530
0x19692490: 0x00000000006e98e0 0x000000001cfd4bd0
0x196924a0: 0x0000001c0000001c 0x0000400700000060
0x196924b0: 0x0000000000000000 0xc00000001f000005
0x196924c0: 0x0000000000000606 0x000000001cfc0004
0x196924d0: 0x0000000005d42ca5 0x000000000000000e
0x196924e0: 0x000000001cfbffe4 0x800000000000000e
0x196924f0: 0x000015554de5d450 0x000015554de5d4a0
0x19692500: 0x000015554df89888 0x000015554df91ec8
0x19692510: 0x000015554dfd0818 0x0000000000003e10
0x19692520: 0x000015554de35288 0x000015554de35258
0x19692530: 0x0000000000000006 0x0000000000003de0
0x19692540: 0x000000001968c50d 0x00000000196924bd
0x19692550: 0x0000000000000000 0x000015554de34540
0x19692560: 0xc00000001f000004 0x0000000000000002
0x19692570: 0x000000001e807624 0x00000000028d4475
0x19692580: 0x0000000000000006 0x8000000000000001
0x19692590: 0x000015554e3190c8 0x0000000019690590

This means that 0x15554efc8fac is most likely the string containing the
bytecode; could you please run

   p *(struct Lisp_String *)0x15554efc8fa8

to see whether this is true, and whether any data is still there?

It would be interesting to know whether we were marking the string or
the bytecode object when the crash happened; please run

   p mark_stk
   p mark_stk.stack[0]
   p mark_stk.stack[1]
   p mark_stk.stack[2]

so we see how deep in the mark stack we were at the time.

(gdb) p *(struct Lisp_String *)0x15554efc8fa8
$9 = {u = {s = {size = 8, size_byte = -3, intervals = 0x0, data = 0x15554f6b661a "\300"...}, next = 0x8, gcaligned = 8 '\b'}}
(gdb)    p mark_stk
$10 = {stack = 0x155539361010, size = 12288, sp = 1}
(gdb)    p mark_stk.stack[0]
$11 = {n = 2, u = {value = 0x196922d0, values = 0x196922d0}}
(gdb)    p mark_stk.stack[1]
$12 = {n = 0, u = {value = 0x6851e793, values = 0x6851e793}}
(gdb)    p mark_stk.stack[2]
$13 = {n = 0, u = {value = 0x160c2d5, values = 0x160c2d5}}