Subject: Re: bug#70423: 29.3; cc-mode hang at 100% cpu and consuming all available memory

From: [email-masked]
Date: Tue, 16 Apr 2024 19:51:09 +0000

Greetings!

I have been able to replicate a (relatively) minimum environment for this — hyperbole + clang-format.el.
I think lsp-mode aggravates the issue somehow, but is not necessary to enter the infinite loop. Without lsp-mode, however, the memory usage seems flat, as opposed to ballooning when it is present.
I would also guess that if you are able to plug in that regexp into cc-mode without installing all of hyperbole.el, you would probably get the same effect, but I didn't know how to do that. 

You will need gzip and the clang-format utility for this from your linux distribution or compiled from source. Preferably version 18.1.3 of the llvm suite, but it might work with earlier versions. I don't think /how/ it's formatting the file matters as much as the fact of it being altered to some degree.

Please follow these steps:
1. start with an empty directory
$ export MYDEBUGDIR=emacs-debugging
2. use the attached init.el file as follows:
$ emacs-29.3 --init-directory=${MYDEBUGDIR} -l init.el
3. Wait for the script to reach the point where it's in an infinite loop (check top/htop for emacs with 100% usage, maybe wait a few seconds or minutes to make sure it is not just taking a long time to run the formatting) and press C-g to send "quit".

This results in a hang and the debugger should show something like this:

Debugger entered--Lisp error: (quit)
  #f(compiled-function () #<bytecode -0x7e8f9110f9b66d2>)()
  syntax-ppss()
  beginning-of-defun-raw(nil)
  beginning-of-defun()
  c-get-fallback-scan-pos(18908)
  c-parse-state-get-strategy(18908 1)
  c-parse-state-1()
  c-parse-state()
  c-laomib-invalidate-cache(18908 18910)
  c-before-change(18908 18910)
  clang-format--replace(18908 2 "\n")
  apply(clang-format--replace (18908 2 "\n"))
  clang-format-region(1 18910 nil nil)
  clang-format-buffer()
  load-with-code-conversion("/home/wrath/emacs-debugging2/init.el" "/home/wrath/emacs-debugging2/init.el" nil t)
  command-line-1(("-l" "emacs-debugging2/init.el"))
  command-line()
  normal-top-level()

or maybe you will see:

Debugger entered--Lisp error: (quit)
  re-search-backward("^\\s(\\|\\(?:^[ \11]*\\(template\\s-*<[^>;.{}]+>\\s-*\\)?\\(..." nil move 1)
  beginning-of-defun-raw(nil)
  beginning-of-defun()
  c-get-fallback-scan-pos(18908)
  c-parse-state-get-strategy(18908 1)
  c-parse-state-1()
  c-parse-state()
  c-laomib-invalidate-cache(18908 18910)
  c-before-change(18908 18910)
  clang-format--replace(18908 2 "\n")
  apply(clang-format--replace (18908 2 "\n"))
  clang-format-region(1 18910 nil nil)
  clang-format-buffer()
  load-with-code-conversion("/home/wrath/emacs-debugging2/init.el" "/home/wrath/emacs-debugging2/init.el" nil t)
  command-line-1(("-l" "emacs-debugging2/init.el"))
  command-line()
  normal-top-level()

Either way it does seem like the problem lies somewhere in cc-mode choking either on the file itself or on that regular expression.

I license the attached init.el under the terms of the AGPL-3 or later version. The drm_formats.cpp file is part of the MIR Server project and is licensed under the terms of the GPL-v3. Both licenses attached. I'm not sure if I am allowed to re-license the MIR file as AGPL, but I believe they are in any case compatible to be distributed together.

Thank you for having a look at this issue, it has been plaguing me since november and even making rust look like an ever so slightly more pleasant alternative to dealing with constant hangs/crashes and having to restore frame/window layouts when editing C/C++..

Kind regards,
M.

On Tuesday, April 16th, 2024 at 7:59 PM, Alan Mackenzie <[email-masked]> wrote:

> Hello.

> Thanks for taking the trouble to submit this bug report.
> 

> On Tue, Apr 16, 2024 at 13:36:16 +0000, 1stmil.eth wrote:
> 

> > Among other scenarios, when clang-format runs as a hook on file save
> > when I make any (even noop) edit in
> > https://github.com/pillowtrucker/mir/blob/main/src/platform/graphics/drm_formats.cpp
> > this file, for example, emacs enters an infinite loop consuming 100%
> > of a cpu core and all memory up to gc threshold.
> 

> > It is not always the regex function on the top of the stack but it is always a call around those functions
> > "beginning-of-defun-raw"
> > "beginning-of-defun"
> > "c-get-fallback-scan-pos"
> > (gdb) bt full
> 

> Yes. c-parse-state is known for sometimes looping - it is an awkward
> spot where doing something rigorusly is likely to be (far) too slow, but
> the ad hoc speed ups occasionally go wrong. That seems to be the
> problem here.
> 

> [ The gdb backtrace (snipped) unfortunately isn't very helpful, here. ]
> 

> > Inside of emacs I get something like this:
> 

> > Debugger entered--Lisp error: (quit) re-search-backward("^\\s(\\|\\(?:^[ \11]\\(template\\s-<[^>;.{}]+>\\s-\\)?\\(\\(\\(auto\\|const\\|explicit\\|extern\\s-+\"[^\"]+\"\\|extern\\|friend\\|inline\\|mutable\\|overload\\|register\\|static\\|typedef\\|virtual\\)\\s-+\\)\\(\\([[<a-zA-Z][]a-zA-Z0-9]\\(::[]_a-zA-Z0-9]+\\)?\\s-<[<>a-zA-Z0-9 ,]+>\\s-[&]\\|[[<a-zA-Z][]_<>a-zA-Z0-9]\\(::[[<a-zA-Z][]<>a-zA-Z0-9]+\\)?\\s-[&]\\)[& \11\n\15]+\\)\\)?\\(\\(::\\|[[<a-zA-Z][]a-zA-Z0-9]\\s-<[^>;{}]+>\\s-[&]*::\\|[[<a-zA-Z][]~<>a-zA-Z0-9]\\s-[&]::\\)\\s-\\)?\\(operator\\s-[^ \11\n\15:;.,?~{}]+\\(\\s-*\\[\\]\\)?\\|[~<a-zA-Z][^][ \11:;.,~{}()\177]\\|[&]?\\([~<a-zA-Z][a-zA-Z0-9]\\s-<[^>;{}]+[ \11\n\15>]*>\\|[~<a-zA-Z][~<>a-zA-Z0-9]\\)\\)\\s-\\(([^{;])\\(\\(\\s-+const\\|\\s-+mutable\\)?\\(\\s-[=:][^;{]+\\)?\\)?\\)\\s-*\\)\\s(" nil move 1)
> > beginning-of-defun-raw(nil)
> > beginning-of-defun()
> > c-get-fallback-scan-pos(18875)
> > c-parse-state-get-strategy(18875 13474)
> > c-parse-state-1()
> > c-parse-state()
> > c-laomib-invalidate-cache(18875 18880)
> > c-before-change(18875 18880)
> 

> The functions thus far suggest that your file is somewhat unusual; it
> has, perhaps, long stretches of text without braces, or long brace
> blocks.
> 

> Could you possibly send me a copy of the above C++ file, or point me to
> an open website (not github) where I could download a copy? I have no
> access to github, since I am not prepared to let probably hostile
> scripts controlled by Microsoft run on my machine.
> 

> > lsp--apply-text-edit-replace-buffer-contents(#<hash-table equal 2/2 0x120ad92d>)
> > #f(compiled-function (edit) #<bytecode -0x9b8e84b9515e2ea>)(#<hash-table equal 2/2 0x120ad92d>)
> > lsp--apply-text-edits((#<hash-table equal 2/2 0x12062b03>) format)
> > lsp-format-buffer()
> > run-hooks(before-save-hook)
> > basic-save-buffer(t)
> > save-buffer(1)
> > funcall-interactively(save-buffer 1)
> > #<subr call-interactively>(save-buffer nil nil)
> > call-interactively@ido-cr+-record-current-command(#<subr call-interactively> save-buffer nil nil)
> > apply(call-interactively@ido-cr+-record-current-command #<subr call-interactively> (save-buffer nil nil))
> > call-interactively(save-buffer nil nil)
> > command-execute(save-buffer)
> 

> > This is not the only scenario when this happens. I can also sometimes
> > trigger the same thing by trying to jump to the definition of a symbol
> > from a .cpp/c++-header file into a system header (all of which are on
> > a read-only filesystem on my computer), for example one of the SDL
> > headers. The main difference is that the buffer-switching gets mixed
> > into the lot and emacs doesn't even respond to C-g and I have to kill
> > -SIGUSR2 it to get it to start responding again, but I'm pretty sure
> > it's the same bug - output from this scenario is here:
> > https://0x0.st/X-RT.txt
> 

> What would be most helpful here would be a reliable recipe to reproduce
> the bug without needing other libraries such as lsp, preferably starting
> from emacs -Q.
> 

> > I'm sorry about the <optimized out> everywhere and lack of inlined
> > sources, but I've already rebuilt it with nixos's enableDebugging
> > flag, which should have enabled -Og and -g3 (I think it did do the
> > latter but failed on the optimizations and left it at O2) and I've
> > already spent most of my day debugging this and it would take another
> > full day to figure out how to properly override the byzantine nixpkgs
> > emacs infrastructure with the right optimization flag and to store the
> > right source tree.
> 

> Yes, bugs are like that sometimes. I'm afraid I don't know anything
> about nixos.
> 

> > In GNU Emacs 29.3 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
> > version 1.18.0, Xaw3d scroll bars)
> > Windowing system distributor 'The X.Org Foundation', version 11.0.12101012
> > System Description: NixOS 24.05 (Uakari)
> 

> [ .... ]

> --
> Alan Mackenzie (Nuremberg, Germany).