Package: emacs;
Reported by: rehan malak <rehan.malak <at> gmail.com>
Date: Sun, 8 Dec 2024 15:01:02 UTC
Severity: normal
Found in version 31.0.50
Done: Stefan Monnier <monnier <at> iro.umontreal.ca>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74738 in the body.
You can then email your comments to 74738 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Sun, 08 Dec 2024 15:01:02 GMT) Full text and rfc822 format available.rehan malak <rehan.malak <at> gmail.com>:bug-gnu-emacs <at> gnu.org.
(Sun, 08 Dec 2024 15:01:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: rehan malak <rehan.malak <at> gmail.com> To: bug-gnu-emacs <at> gnu.org Subject: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Sun, 8 Dec 2024 14:05:36 +0100
[Message part 1 (text/plain, inline)]
Hi,
I can reproduce systematically this freeze dealing with a 10000lines
Python file :
wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
emacs -Q gef.py
PageDown several times or scrolling with the mouse
Emacs freezes, Ctrl-g not working, CPU 100%
It works also by searching : Ctrl-s show RET
then Ctrl-s several times
With a minimal .emacs :
(set debug-on-error t)
(set debug-on-quit t)
and before scrolling
M-x profiler-start RET
then in an external terminal
pkill -SIGUSR2 emacs
I get the backtrace :
Debugger entered--entering a function:
* #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
syntax-ppss()
python-syntax-context-type()
python-nav-forward-block(-1)
python-nav-backward-block()
python-nav-beginning-of-block()
python-nav-end-of-block()
python-info-statement-ends-block-p()
python-info-end-of-block-p()
python-nav--forward-sexp(1 nil nil)
python-nav-forward-sexp(1)
forward-sexp(1)
up-list(1)
python--font-lock-f-strings(30419)
font-lock-fontify-keywords-region(28914 30419 nil)
font-lock-default-fontify-region(28914 30414 nil)
font-lock-fontify-region(28914 30414)
#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>)(font-lock-fontify-region)
run-hook-wrapped(#f(compiled-function (fun) #<bytecode
-0x8a96d65e1315e71>) font-lock-fontify-region)
jit-lock--run-functions(28914 30414)
jit-lock-fontify-now(28914 30414)
jit-lock-function(28914)
and can look at profiler report
M-x profiler-stop
M-x profiler-report
showing time spent in syntax-ppss
17770 88% - jit-lock-function
17770 88% - jit-lock-fontify-now
17770 88% - jit-lock--run-functions
17770 88% - run-hook-wrapped
17770 88% - #<byte-code-function FA0>
17770 88% - font-lock-fontify-region
17770 88% - font-lock-default-fontify-region
17690 87% - font-lock-fontify-keywords-region
17678 87% - python--font-lock-f-strings
17674 87% - up-list
17674 87% - forward-sexp
17674 87% - python-nav-forward-sexp
17674 87% - python-nav--forward-sexp
9253 45% - python-info-end-of-block-p
9245 45% - python-info-statement-ends-block-p
9221 45% - python-nav-end-of-block
8609 42% - python-nav-beginning-of-block
8557 42% - python-nav-backward-block
8557 42% - python-nav-forward-block
8005 39% - python-syntax-context-type
7997 39% - syntax-ppss
7097 35% parse-partial-sexp
753 3% + #<byte-code-function A23>
376 1% + python-nav-beginning-of-statement
132 0% re-search-backward
24 0% looking-at
28 0% current-indentation
8 0% + python-nav-beginning-of-statement
4 0% python-info-current-line-empty-p
4 0% + python-info-current-line-comment-p
604 2% + python-nav-end-of-statement
4 0% + python-info-current-line-comment-p
4 0% current-indentation
24 0% + python-nav-end-of-statement
8 0% + python-info-end-of-statement-p
8178 40% - python-info-statement-ends-block-p
8150 40% - python-nav-end-of-block
7754 38% - python-nav-beginning-of-block
7738 38% - python-nav-backward-block
7734 38% - python-nav-forward-block
7178 35% - python-syntax-context-type
7174 35% - syntax-ppss
6978 34% parse-partial-sexp
8 0% make-closure
8 0% #<byte-code-function B1E>
4 0% syntax-table
4 0% syntax-ppss--update-stats
412 2% + python-nav-beginning-of-statement
92 0% re-search-backward
44 0% looking-at
12 0% current-indentation
4 0% looking-at
380 1% + python-nav-end-of-statement
16 0% + python-util-forward-comment
28 0% + python-nav-end-of-statement
151 0% + python-nav-end-of-block
28 0% + python-info-beginning-of-block-p
24 0% + python-info-end-of-statement-p
20 0% + python-info-beginning-of-statement-p
8 0% + python-syntax-context-type
8 0% + python-info-statement-starts-block-p
4 0% + syntax-ppss
8 0% + #<byte-code-function 7E0>
4 0% + #<byte-code-function 740>
80 0% + font-lock-fontify-syntactically-region
2279 11% Automatic GC
73 0% + redisplay_internal (C function)
40 0% + command-execute
8 0% + ...
4 0% + mouse--click-1-maybe-follows-link
other people talking about python/profiling/syntax-ppss :
https://www.reddit.com/r/emacs/comments/z0oye9/emacs_freezes_when_opening_a_python_file/
https://www.reddit.com/r/emacs/comments/9h7onq/large_python_files_hanging_with_an_unmatched_quote/
Please tell me how I can help ?
Please tell me if there is a workaround without losing python syntax color ?
Thanks
Rehan
In GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.43, cairo version 1.18.2) of 2024-12-08 built on cndl
Repository revision: b953bcb17047998c9e41cede7c5e5ffec22209b2
Repository branch: master
System Description: Debian GNU/Linux trixie/sid
Configured using:
'configure --prefix=/home/jean/local/emacs --with-pgtk'
Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX
LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PGTK PNG SECCOMP SOUND THREADS
TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3 ZLIB
Important settings:
value of $LANG: en_US.UTF-8
value of $XMODIFIERS: @im=ibus
locale-coding-system: utf-8-unix
Major mode: Python
Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
minibuffer-regexp-mode: t
line-number-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
Load-path shadows:
None found.
Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-seq python
rx project byte-opt gv bytecomp byte-compile compat pcase treesit comint
subr-x ansi-osc ring cl-loaddefs cl-lib ansi-color rmc iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
touch-screen pgtk-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify dynamic-setting system-font-setting
font-render-setting cairo gtk pgtk multi-tty move-toolbar
make-network-process emacs)
Memory information:
((conses 16 60995 13527) (symbols 48 7517 0) (strings 32 19122 1493)
(string-bytes 1 515162) (vectors 16 13554)
(vector-slots 8 145472 8433) (floats 8 26 2) (intervals 56 436 0)
(buffers 984 12))
[Message part 2 (text/html, inline)]
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Sun, 08 Dec 2024 17:38:02 GMT) Full text and rfc822 format available.Message #8 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: rehan malak <rehan.malak <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>, kobarity <kobarity <at> gmail.com> Cc: 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Sun, 08 Dec 2024 19:37:02 +0200
> From: rehan malak <rehan.malak <at> gmail.com>
> Date: Sun, 8 Dec 2024 14:05:36 +0100
>
> I can reproduce systematically this freeze dealing with a 10000lines
> Python file :
>
> wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> emacs -Q gef.py
A much smaller reproducer is attached below. Just visiting it freezes
Emacs. But if I change the line marked below:
@classmethod
def is_valid(cls, _: pathlib.Path) -> bool:
raise NotImplementedError
def __str__(self) -> str:
>>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
to say this instead:
return self.name
the problem goes away. So it is something in that complex expression
that trips syntax-ppss.
Stefan and kobarity, any suggestions or ideas?
> PageDown several times or scrolling with the mouse
>
> Emacs freezes, Ctrl-g not working, CPU 100%
>
> It works also by searching : Ctrl-s show RET
> then Ctrl-s several times
>
> With a minimal .emacs :
>
> (set debug-on-error t)
> (set debug-on-quit t)
>
> and before scrolling
>
> M-x profiler-start RET
>
> then in an external terminal
>
> pkill -SIGUSR2 emacs
>
> I get the backtrace :
>
> Debugger entered--entering a function:
> * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
> syntax-ppss()
> python-syntax-context-type()
> python-nav-forward-block(-1)
> python-nav-backward-block()
> python-nav-beginning-of-block()
> python-nav-end-of-block()
> python-info-statement-ends-block-p()
> python-info-end-of-block-p()
> python-nav--forward-sexp(1 nil nil)
> python-nav-forward-sexp(1)
> forward-sexp(1)
> up-list(1)
> python--font-lock-f-strings(30419)
> font-lock-fontify-keywords-region(28914 30419 nil)
> font-lock-default-fontify-region(28914 30414 nil)
> font-lock-fontify-region(28914 30414)
> #f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>)(font-lock-fontify-region)
> run-hook-wrapped(#f(compiled-function (fun) #<bytecode -0x8a96d65e1315e71>) font-lock-fontify-region)
> jit-lock--run-functions(28914 30414)
> jit-lock-fontify-now(28914 30414)
> jit-lock-function(28914)
I see something different, in GDB:
Lisp Backtrace:
"parse-partial-sexp" (0x9f1a788)
"syntax-ppss" (0x9f1a710)
"python-info-line-ends-backslash-p" (0x9f1a6b8)
"python-nav-end-of-statement" (0x9f1a670)
"python-nav-end-of-block" (0x9f1a630)
"python-info-statement-ends-block-p" (0x9f1a610)
"python-info-end-of-block-p" (0x9f1a5c0)
"python-nav--forward-sexp" (0x9f1a548)
"python-nav-forward-sexp" (0x9f1a4f8)
"forward-sexp" (0x9f1a4a8)
"up-list" (0x9f1a438)
"python--font-lock-f-strings" (0x9f1a3a0)
"font-lock-fontify-keywords-region" (0x9f1a308)
"font-lock-default-fontify-region" (0x9f1a2a0)
"font-lock-fontify-region" (0x9f1a230)
0xb9117d0 PVEC_CLOSURE
"run-hook-wrapped" (0x9f1a1c0)
"jit-lock--run-functions" (0x9f1a0e8)
"jit-lock-fontify-now" (0x9f1a058)
"jit-lock-function" (0x5ffba98)
"redisplay_internal (C function)" (0x0)
Here's the file with which I can reproduce the problem?
class FileFormat:
name: str
path: pathlib.Path
entry_point: int
checksec: dict[str, bool]
sections: list[FileFormatSection]
def __init__(self, path: str | pathlib.Path) -> None:
raise NotImplementedError
def __init_subclass__(cls: Type["FileFormat"], **kwargs):
global __registered_file_formats__
super().__init_subclass__(**kwargs)
required_attributes = ("name", "entry_point", "is_valid", "checksec",)
for attr in required_attributes:
if not hasattr(cls, attr):
raise NotImplementedError(f"File format '{cls.__name__}' is invalid: missing attribute '{attr}'")
__registered_file_formats__.add(cls)
return
@classmethod
def is_valid(cls, _: pathlib.Path) -> bool:
raise NotImplementedError
def __str__(self) -> str:
return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
class Elf(FileFormat):
"""Basic ELF parsing.
Ref:
- http://www.skyfree.org/linux/references/ELF_Format.pdf
- https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf
- https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html
"""
class Class(enum.Enum):
ELF_32_BITS = 0x01
ELF_64_BITS = 0x02
ELF_MAGIC = 0x7f454c46
class Abi(enum.Enum):
X86_64 = 0x3e
X86_32 = 0x03
ARM = 0x28
MIPS = 0x08
POWERPC = 0x14
POWERPC64 = 0x15
SPARC = 0x02
SPARC64 = 0x2b
AARCH64 = 0xb7
RISCV = 0xf3
IA64 = 0x32
M68K = 0x04
class Type(enum.Enum):
ET_RELOC = 1
ET_EXEC = 2
ET_DYN = 3
ET_CORE = 4
class OsAbi(enum.Enum):
SYSTEMV = 0x00
HPUX = 0x01
NETBSD = 0x02
LINUX = 0x03
SOLARIS = 0x06
AIX = 0x07
IRIX = 0x08
FREEBSD = 0x09
OPENBSD = 0x0C
e_magic: int = ELF_MAGIC
e_class: "Elf.Class" = Class.ELF_32_BITS
e_endianness: Endianness = Endianness.LITTLE_ENDIAN
e_eiversion: int
e_osabi: "Elf.OsAbi"
e_abiversion: int
e_pad: bytes
e_type: "Elf.Type" = Type.ET_EXEC
e_machine: Abi = Abi.X86_32
e_version: int
e_entry: int
e_phoff: int
e_shoff: int
e_flags: int
e_ehsize: int
e_phentsize: int
e_phnum: int
e_shentsize: int
e_shnum: int
e_shstrndx: int
path: pathlib.Path
phdrs : list["Phdr"]
shdrs : list["Shdr"]
name: str = "ELF"
__checksec : dict[str, bool]
def __init__(self, path: str | pathlib.Path) -> None:
"""Instantiate an ELF object. A valid ELF must be provided, or an exception will be thrown."""
if isinstance(path, str):
self.path = pathlib.Path(path).expanduser()
elif isinstance(path, pathlib.Path):
self.path = path
else:
raise TypeError
if not self.path.exists():
raise FileNotFoundError(f"'{self.path}' not found/readable, most gef features will not work")
self.__checksec = {}
with self.path.open("rb") as self.fd:
# off 0x0
self.e_magic, e_class, e_endianness, self.e_eiversion = self.read_and_unpack(">IBBB")
if self.e_magic != Elf.ELF_MAGIC:
# The ELF is corrupted, GDB won't handle it, no point going further
raise RuntimeError("Not a valid ELF file (magic)")
self.e_class, self.e_endianness = Elf.Class(e_class), Endianness(e_endianness)
if self.e_endianness != gef.arch.endianness:
warn("Unexpected endianness for architecture")
endian = self.e_endianness
# off 0x7
e_osabi, self.e_abiversion = self.read_and_unpack(f"{endian}BB")
self.e_osabi = Elf.OsAbi(e_osabi)
# off 0x9
self.e_pad = self.read(7)
# off 0x10
e_type, e_machine, self.e_version = self.read_and_unpack(f"{endian}HHI")
self.e_type, self.e_machine = Elf.Type(e_type), Elf.Abi(e_machine)
# off 0x18
if self.e_class == Elf.Class.ELF_64_BITS:
self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}QQQ")
else:
self.e_entry, self.e_phoff, self.e_shoff = self.read_and_unpack(f"{endian}III")
self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum = self.read_and_unpack(f"{endian}IHHH")
self.e_shentsize, self.e_shnum, self.e_shstrndx = self.read_and_unpack(f"{endian}HHH")
self.phdrs = []
for i in range(self.e_phnum):
self.phdrs.append(Phdr(self, self.e_phoff + self.e_phentsize * i))
self.shdrs = []
for i in range(self.e_shnum):
self.shdrs.append(Shdr(self, self.e_shoff + self.e_shentsize * i))
return
def read(self, size: int) -> bytes:
return self.fd.read(size)
def read_and_unpack(self, fmt: str) -> tuple[Any, ...]:
size = struct.calcsize(fmt)
data = self.fd.read(size)
return struct.unpack(fmt, data)
def seek(self, off: int) -> None:
self.fd.seek(off, 0)
def __str__(self) -> str:
return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})"
def __repr__(self) -> str:
return f"ELF('{self.path.absolute()}', {self.e_class.name}, {self.e_machine.name})"
@property
def entry_point(self) -> int:
return self.e_entry
@classmethod
def is_valid(cls, path: pathlib.Path) -> bool:
return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) == Elf.ELF_MAGIC
@property
def checksec(self) -> dict[str, bool]:
"""Check the security property of the ELF binary. The following properties are:
- Canary
- NX
- PIE
- Fortify
- Partial/Full RelRO.
Return a dict() with the different keys mentioned above, and the boolean
associated whether the protection was found."""
if not self.__checksec:
def __check_security_property(opt: str, filename: str, pattern: str) -> bool:
cmd = [readelf,]
cmd += opt.split()
cmd += [filename,]
lines = gef_execute_external(cmd, as_list=True)
for line in lines:
if re.search(pattern, line):
return True
return False
abspath = str(self.path.absolute())
readelf = gef.session.constants["readelf"]
self.__checksec["Canary"] = __check_security_property("-rs", abspath, r"__stack_chk_fail") is True
has_gnu_stack = __check_security_property("-W -l", abspath, r"GNU_STACK") is True
if has_gnu_stack:
self.__checksec["NX"] = __check_security_property("-W -l", abspath, r"GNU_STACK.*RWE") is False
else:
self.__checksec["NX"] = False
self.__checksec["PIE"] = __check_security_property("-h", abspath, r":.*EXEC") is False
self.__checksec["Fortify"] = __check_security_property("-s", abspath, r"_chk <at> GLIBC") is True
self.__checksec["Partial RelRO"] = __check_security_property("-l", abspath, r"GNU_RELRO") is True
self.__checksec["Full RelRO"] = self.__checksec["Partial RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True
return self.__checksec
@classproperty
@deprecated("use `Elf.Abi.X86_64`")
def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.X86_32`")
def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.ARM`")
def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.MIPS`")
def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.POWERPC`")
def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.POWERPC64`")
def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.SPARC`")
def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.SPARC64`")
def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.AARCH64`")
def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value # pylint: disable=no-self-argument
@classproperty
@deprecated("use `Elf.Abi.RISCV`")
def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint: disable=no-self-argument
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Mon, 09 Dec 2024 15:06:02 GMT) Full text and rfc822 format available.Message #11 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: kobarity <kobarity <at> gmail.com> To: rehan malak <rehan.malak <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org> Cc: 74738 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Tue, 10 Dec 2024 00:04:28 +0900
rehan malak wrote:
> Please tell me if there is a workaround without losing python syntax color ?
If you can configure Emacs to use tree-sitter, python-ts-mode would
not be affected by this problem.
Eli Zaretskii wrote:
>
> > From: rehan malak <rehan.malak <at> gmail.com>
> > Date: Sun, 8 Dec 2024 14:05:36 +0100
> >
> > I can reproduce systematically this freeze dealing with a 10000lines
> > Python file :
> >
> > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> > emacs -Q gef.py
It seems to be reproducible after the following commit:
commit 3b3274a85c2f5df21d76d82e0d7740005aa84fdf
Author: Stefan Monnier <monnier <at> iro.umontreal.ca>
Date: Fri Oct 16 14:03:59 2020 -0400
* lisp/progmodes/python.el: Teach f-strings to `font-lock`
(python--f-string-p, python--font-lock-f-strings): New functions.
(python-font-lock-keywords-maximum-decoration): Use them.
> A much smaller reproducer is attached below. Just visiting it freezes
> Emacs. But if I change the line marked below:
>
> @classmethod
> def is_valid(cls, _: pathlib.Path) -> bool:
> raise NotImplementedError
>
> def __str__(self) -> str:
> >>>>> return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point:#x})"
>
> to say this instead:
>
> return self.name
>
> the problem goes away. So it is something in that complex expression
> that trips syntax-ppss.
>
> Stefan and kobarity, any suggestions or ideas?
I could not reproduce the freeze, but it took several seconds to visit
the reproducer. It appears to be possible to work around this by
simply removing the format specifier ":#x".
return f"{self.name}('{self.path.absolute()}', entry @ {self.entry_point})"
So there may be a problem with font-lock when there is an f-string
format specifier, but I have not found the root cause yet.
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Mon, 09 Dec 2024 15:25:02 GMT) Full text and rfc822 format available.Message #14 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: rehan malak <rehan.malak <at> gmail.com> To: Eli Zaretskii <eliz <at> gnu.org> Cc: kobarity <kobarity <at> gmail.com>, Stefan Monnier <monnier <at> iro.umontreal.ca>, 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Mon, 9 Dec 2024 15:58:22 +0100
[Message part 1 (text/plain, inline)]
Hi,
Deleting all the "#" in the f-string of the gef.py file removes the problem
on my side.
sed -i -e 's/#0{align/0{align/g' gef.py
sed -i -e 's/#0{width/0{width/g' gef.py
sed -i -e 's/#07x}/07x}/g' gef.py
sed -i -e 's/#06x}/06x}/g' gef.py
sed -i -e 's/#04x}/04x}/g' gef.py
sed -i -e 's/#4x}/4x}/g' gef.py
sed -i -e 's/#8x}/8x}/g' gef.py
sed -i -e 's/#10x}/10x}/g' gef.py
sed -i -e 's/#x}/x}/g' gef.py
Your smaller example Eli also contains a f-string with the "#".
This python f-string format specifier is described here:
https://docs.python.org/3/library/string.html#format-specification-mini-language
Is this format specifier supported from the beginning ?
For example :
value = 0xab
print(f"{value:x} is a value")
=> color syntax : *is a value* has the color defined by
font-lock-string-face
while
value = 0xab
print(f"{value:#x} is a value")
=> color are messed up... *is* has color defined by font-lock-keyword-face
and *a value* has color defined by default face
Best,
Rehan
On Sun, Dec 8, 2024 at 6:37 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> > From: rehan malak <rehan.malak <at> gmail.com>
> > Date: Sun, 8 Dec 2024 14:05:36 +0100
> >
> > I can reproduce systematically this freeze dealing with a 10000lines
> > Python file :
> >
> > wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py
> > emacs -Q gef.py
>
> A much smaller reproducer is attached below. Just visiting it freezes
> Emacs. But if I change the line marked below:
>
> @classmethod
> def is_valid(cls, _: pathlib.Path) -> bool:
> raise NotImplementedError
>
> def __str__(self) -> str:
> >>>>> return f"{self.name}('{self.path.absolute()}', entry @
> {self.entry_point:#x})"
>
> to say this instead:
>
> return self.name
>
> the problem goes away. So it is something in that complex expression
> that trips syntax-ppss.
>
> Stefan and kobarity, any suggestions or ideas?
>
> > PageDown several times or scrolling with the mouse
> >
> > Emacs freezes, Ctrl-g not working, CPU 100%
> >
> > It works also by searching : Ctrl-s show RET
> > then Ctrl-s several times
> >
> > With a minimal .emacs :
> >
> > (set debug-on-error t)
> > (set debug-on-quit t)
> >
> > and before scrolling
> >
> > M-x profiler-start RET
> >
> > then in an external terminal
> >
> > pkill -SIGUSR2 emacs
> >
> > I get the backtrace :
> >
> > Debugger entered--entering a function:
> > * #f(compiled-function () #<bytecode -0x179dcd1db31182ae>)()
> > syntax-ppss()
> > python-syntax-context-type()
> > python-nav-forward-block(-1)
> > python-nav-backward-block()
> > python-nav-beginning-of-block()
> > python-nav-end-of-block()
> > python-info-statement-ends-block-p()
> > python-info-end-of-block-p()
> > python-nav--forward-sexp(1 nil nil)
> > python-nav-forward-sexp(1)
> > forward-sexp(1)
> > up-list(1)
> > python--font-lock-f-strings(30419)
> > font-lock-fontify-keywords-region(28914 30419 nil)
> > font-lock-default-fontify-region(28914 30414 nil)
> > font-lock-fontify-region(28914 30414)
> > #f(compiled-function (fun) #<bytecode
> -0x8a96d65e1315e71>)(font-lock-fontify-region)
> > run-hook-wrapped(#f(compiled-function (fun) #<bytecode
> -0x8a96d65e1315e71>) font-lock-fontify-region)
> > jit-lock--run-functions(28914 30414)
> > jit-lock-fontify-now(28914 30414)
> > jit-lock-function(28914)
>
> I see something different, in GDB:
>
> Lisp Backtrace:
> "parse-partial-sexp" (0x9f1a788)
> "syntax-ppss" (0x9f1a710)
> "python-info-line-ends-backslash-p" (0x9f1a6b8)
> "python-nav-end-of-statement" (0x9f1a670)
> "python-nav-end-of-block" (0x9f1a630)
> "python-info-statement-ends-block-p" (0x9f1a610)
> "python-info-end-of-block-p" (0x9f1a5c0)
> "python-nav--forward-sexp" (0x9f1a548)
> "python-nav-forward-sexp" (0x9f1a4f8)
> "forward-sexp" (0x9f1a4a8)
> "up-list" (0x9f1a438)
> "python--font-lock-f-strings" (0x9f1a3a0)
> "font-lock-fontify-keywords-region" (0x9f1a308)
> "font-lock-default-fontify-region" (0x9f1a2a0)
> "font-lock-fontify-region" (0x9f1a230)
> 0xb9117d0 PVEC_CLOSURE
> "run-hook-wrapped" (0x9f1a1c0)
> "jit-lock--run-functions" (0x9f1a0e8)
> "jit-lock-fontify-now" (0x9f1a058)
> "jit-lock-function" (0x5ffba98)
> "redisplay_internal (C function)" (0x0)
>
> Here's the file with which I can reproduce the problem?
>
>
> class FileFormat:
> name: str
> path: pathlib.Path
> entry_point: int
> checksec: dict[str, bool]
> sections: list[FileFormatSection]
>
> def __init__(self, path: str | pathlib.Path) -> None:
> raise NotImplementedError
>
> def __init_subclass__(cls: Type["FileFormat"], **kwargs):
> global __registered_file_formats__
> super().__init_subclass__(**kwargs)
> required_attributes = ("name", "entry_point", "is_valid",
> "checksec",)
> for attr in required_attributes:
> if not hasattr(cls, attr):
> raise NotImplementedError(f"File format '{cls.__name__}'
> is invalid: missing attribute '{attr}'")
> __registered_file_formats__.add(cls)
> return
>
> @classmethod
> def is_valid(cls, _: pathlib.Path) -> bool:
> raise NotImplementedError
>
> def __str__(self) -> str:
> return f"{self.name}('{self.path.absolute()}', entry @
> {self.entry_point:#x})"
>
>
> class Elf(FileFormat):
> """Basic ELF parsing.
> Ref:
> - http://www.skyfree.org/linux/references/ELF_Format.pdf
> - https://refspecs.linuxfoundation.org/elf/elfspec_ppc.pdf
> - https://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi.html
> """
> class Class(enum.Enum):
> ELF_32_BITS = 0x01
> ELF_64_BITS = 0x02
>
> ELF_MAGIC = 0x7f454c46
>
> class Abi(enum.Enum):
> X86_64 = 0x3e
> X86_32 = 0x03
> ARM = 0x28
> MIPS = 0x08
> POWERPC = 0x14
> POWERPC64 = 0x15
> SPARC = 0x02
> SPARC64 = 0x2b
> AARCH64 = 0xb7
> RISCV = 0xf3
> IA64 = 0x32
> M68K = 0x04
>
> class Type(enum.Enum):
> ET_RELOC = 1
> ET_EXEC = 2
> ET_DYN = 3
> ET_CORE = 4
>
> class OsAbi(enum.Enum):
> SYSTEMV = 0x00
> HPUX = 0x01
> NETBSD = 0x02
> LINUX = 0x03
> SOLARIS = 0x06
> AIX = 0x07
> IRIX = 0x08
> FREEBSD = 0x09
> OPENBSD = 0x0C
>
> e_magic: int = ELF_MAGIC
> e_class: "Elf.Class" = Class.ELF_32_BITS
> e_endianness: Endianness = Endianness.LITTLE_ENDIAN
> e_eiversion: int
> e_osabi: "Elf.OsAbi"
> e_abiversion: int
> e_pad: bytes
> e_type: "Elf.Type" = Type.ET_EXEC
> e_machine: Abi = Abi.X86_32
> e_version: int
> e_entry: int
> e_phoff: int
> e_shoff: int
> e_flags: int
> e_ehsize: int
> e_phentsize: int
> e_phnum: int
> e_shentsize: int
> e_shnum: int
> e_shstrndx: int
>
> path: pathlib.Path
> phdrs : list["Phdr"]
> shdrs : list["Shdr"]
> name: str = "ELF"
>
> __checksec : dict[str, bool]
>
> def __init__(self, path: str | pathlib.Path) -> None:
> """Instantiate an ELF object. A valid ELF must be provided, or an
> exception will be thrown."""
>
> if isinstance(path, str):
> self.path = pathlib.Path(path).expanduser()
> elif isinstance(path, pathlib.Path):
> self.path = path
> else:
> raise TypeError
>
> if not self.path.exists():
> raise FileNotFoundError(f"'{self.path}' not found/readable,
> most gef features will not work")
>
> self.__checksec = {}
>
> with self.path.open("rb") as self.fd:
> # off 0x0
> self.e_magic, e_class, e_endianness, self.e_eiversion =
> self.read_and_unpack(">IBBB")
> if self.e_magic != Elf.ELF_MAGIC:
> # The ELF is corrupted, GDB won't handle it, no point
> going further
> raise RuntimeError("Not a valid ELF file (magic)")
>
> self.e_class, self.e_endianness = Elf.Class(e_class),
> Endianness(e_endianness)
>
> if self.e_endianness != gef.arch.endianness:
> warn("Unexpected endianness for architecture")
>
> endian = self.e_endianness
>
> # off 0x7
> e_osabi, self.e_abiversion =
> self.read_and_unpack(f"{endian}BB")
> self.e_osabi = Elf.OsAbi(e_osabi)
>
> # off 0x9
> self.e_pad = self.read(7)
>
> # off 0x10
> e_type, e_machine, self.e_version =
> self.read_and_unpack(f"{endian}HHI")
> self.e_type, self.e_machine = Elf.Type(e_type),
> Elf.Abi(e_machine)
>
> # off 0x18
> if self.e_class == Elf.Class.ELF_64_BITS:
> self.e_entry, self.e_phoff, self.e_shoff =
> self.read_and_unpack(f"{endian}QQQ")
> else:
> self.e_entry, self.e_phoff, self.e_shoff =
> self.read_and_unpack(f"{endian}III")
>
> self.e_flags, self.e_ehsize, self.e_phentsize, self.e_phnum =
> self.read_and_unpack(f"{endian}IHHH")
> self.e_shentsize, self.e_shnum, self.e_shstrndx =
> self.read_and_unpack(f"{endian}HHH")
>
> self.phdrs = []
> for i in range(self.e_phnum):
> self.phdrs.append(Phdr(self, self.e_phoff +
> self.e_phentsize * i))
>
> self.shdrs = []
> for i in range(self.e_shnum):
> self.shdrs.append(Shdr(self, self.e_shoff +
> self.e_shentsize * i))
> return
>
> def read(self, size: int) -> bytes:
> return self.fd.read(size)
>
> def read_and_unpack(self, fmt: str) -> tuple[Any, ...]:
> size = struct.calcsize(fmt)
> data = self.fd.read(size)
> return struct.unpack(fmt, data)
>
> def seek(self, off: int) -> None:
> self.fd.seek(off, 0)
>
> def __str__(self) -> str:
> return f"ELF('{self.path.absolute()}', {self.e_class.name}, {
> self.e_machine.name})"
>
> def __repr__(self) -> str:
> return f"ELF('{self.path.absolute()}', {self.e_class.name}, {
> self.e_machine.name})"
>
> @property
> def entry_point(self) -> int:
> return self.e_entry
>
> @classmethod
> def is_valid(cls, path: pathlib.Path) -> bool:
> return u32(path.open("rb").read(4), e = Endianness.BIG_ENDIAN) ==
> Elf.ELF_MAGIC
>
> @property
> def checksec(self) -> dict[str, bool]:
> """Check the security property of the ELF binary. The following
> properties are:
> - Canary
> - NX
> - PIE
> - Fortify
> - Partial/Full RelRO.
> Return a dict() with the different keys mentioned above, and the
> boolean
> associated whether the protection was found."""
> if not self.__checksec:
> def __check_security_property(opt: str, filename: str,
> pattern: str) -> bool:
> cmd = [readelf,]
> cmd += opt.split()
> cmd += [filename,]
> lines = gef_execute_external(cmd, as_list=True)
> for line in lines:
> if re.search(pattern, line):
> return True
> return False
>
> abspath = str(self.path.absolute())
> readelf = gef.session.constants["readelf"]
> self.__checksec["Canary"] = __check_security_property("-rs",
> abspath, r"__stack_chk_fail") is True
> has_gnu_stack = __check_security_property("-W -l", abspath,
> r"GNU_STACK") is True
> if has_gnu_stack:
> self.__checksec["NX"] = __check_security_property("-W -l",
> abspath, r"GNU_STACK.*RWE") is False
> else:
> self.__checksec["NX"] = False
> self.__checksec["PIE"] = __check_security_property("-h",
> abspath, r":.*EXEC") is False
> self.__checksec["Fortify"] = __check_security_property("-s",
> abspath, r"_chk <at> GLIBC") is True
> self.__checksec["Partial RelRO"] =
> __check_security_property("-l", abspath, r"GNU_RELRO") is True
> self.__checksec["Full RelRO"] = self.__checksec["Partial
> RelRO"] and __check_security_property("-d", abspath, r"BIND_NOW") is True
> return self.__checksec
>
> @classproperty
> @deprecated("use `Elf.Abi.X86_64`")
> def X86_64(cls) -> int: return Elf.Abi.X86_64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.X86_32`")
> def X86_32(cls) -> int : return Elf.Abi.X86_32.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.ARM`")
> def ARM(cls) -> int : return Elf.Abi.ARM.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.MIPS`")
> def MIPS(cls) -> int : return Elf.Abi.MIPS.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.POWERPC`")
> def POWERPC(cls) -> int : return Elf.Abi.POWERPC.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.POWERPC64`")
> def POWERPC64(cls) -> int : return Elf.Abi.POWERPC64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.SPARC`")
> def SPARC(cls) -> int : return Elf.Abi.SPARC.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.SPARC64`")
> def SPARC64(cls) -> int : return Elf.Abi.SPARC64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.AARCH64`")
> def AARCH64(cls) -> int : return Elf.Abi.AARCH64.value # pylint:
> disable=no-self-argument
>
> @classproperty
> @deprecated("use `Elf.Abi.RISCV`")
> def RISCV(cls) -> int : return Elf.Abi.RISCV.value # pylint:
> disable=no-self-argument
>
>
[Message part 2 (text/html, inline)]
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Wed, 11 Dec 2024 14:26:02 GMT) Full text and rfc822 format available.Message #17 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: kobarity <kobarity <at> gmail.com> To: rehan malak <rehan.malak <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org> Cc: 74738 <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca> Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Wed, 11 Dec 2024 23:24:34 +0900
[Message part 1 (text/plain, inline)]
rehan malak wrote:
> Deleting all the "#" in the f-string of the gef.py file removes the problem on my side.
> sed -i -e 's/#0{align/0{align/g' gef.py
> sed -i -e 's/#0{width/0{width/g' gef.py
> sed -i -e 's/#07x}/07x}/g' gef.py
> sed -i -e 's/#06x}/06x}/g' gef.py
> sed -i -e 's/#04x}/04x}/g' gef.py
> sed -i -e 's/#4x}/4x}/g' gef.py
> sed -i -e 's/#8x}/8x}/g' gef.py
> sed -i -e 's/#10x}/10x}/g' gef.py
> sed -i -e 's/#x}/x}/g' gef.py
>
> Your smaller example Eli also contains a f-string with the "#".
>
> This python f-string format specifier is described here:
> https://docs.python.org/3/library/string.html#format-specification-mini-language
>
> Is this format specifier supported from the beginning ?
>
> For example :
>
> value = 0xab
> print(f"{value:x} is a value")
>
> => color syntax : is a value has the color defined by font-lock-string-face
>
> while
>
> value = 0xab
> print(f"{value:#x} is a value")
>
> => color are messed up... is has color defined by font-lock-keyword-face and a value has color defined by default face
The mechanism of the hang is still unclear, but I found the cause of
the font-lock malfunction: when using `up-list' to find the
corresponding closing brace for an opening brace in an f-string, if
the format specifier contains "#", the rest of the string is
considered a comment and the search fails. Therefore, it can be
worked around by temporarily binding `parse-sexp-ignore-comments' to
nil, as in the attached patch.
Hangs on gef.py can also be avoided with this patch, but it assumes
that the braces are properly closed. If you remove the closing braces
of expressions containing "#", it will eventually hang.
[0001-Fix-font-lock-of-Python-f-strings.patch (text/plain, attachment)]
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Thu, 12 Dec 2024 03:50:02 GMT) Full text and rfc822 format available.Message #20 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: rehan malak <rehan.malak <at> gmail.com> To: kobarity <kobarity <at> gmail.com> Cc: Eli Zaretskii <eliz <at> gnu.org>, Stefan Monnier <monnier <at> iro.umontreal.ca>, 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Thu, 12 Dec 2024 04:49:34 +0100
[Message part 1 (text/plain, inline)]
thanks kobarity, with your patch, I confirm that :
- I can open gef.py and scroll
- I get the correct font-lock-string-face in the f-string after a "#" in
the format
On Wed, Dec 11, 2024 at 3:24 PM kobarity <kobarity <at> gmail.com> wrote:
> rehan malak wrote:
> > Deleting all the "#" in the f-string of the gef.py file removes the
> problem on my side.
> > sed -i -e 's/#0{align/0{align/g' gef.py
> > sed -i -e 's/#0{width/0{width/g' gef.py
> > sed -i -e 's/#07x}/07x}/g' gef.py
> > sed -i -e 's/#06x}/06x}/g' gef.py
> > sed -i -e 's/#04x}/04x}/g' gef.py
> > sed -i -e 's/#4x}/4x}/g' gef.py
> > sed -i -e 's/#8x}/8x}/g' gef.py
> > sed -i -e 's/#10x}/10x}/g' gef.py
> > sed -i -e 's/#x}/x}/g' gef.py
> >
> > Your smaller example Eli also contains a f-string with the "#".
> >
> > This python f-string format specifier is described here:
> >
> https://docs.python.org/3/library/string.html#format-specification-mini-language
> >
> > Is this format specifier supported from the beginning ?
> >
> > For example :
> >
> > value = 0xab
> > print(f"{value:x} is a value")
> >
> > => color syntax : is a value has the color defined by
> font-lock-string-face
> >
> > while
> >
> > value = 0xab
> > print(f"{value:#x} is a value")
> >
> > => color are messed up... is has color defined by font-lock-keyword-face
> and a value has color defined by default face
>
> The mechanism of the hang is still unclear, but I found the cause of
> the font-lock malfunction: when using `up-list' to find the
> corresponding closing brace for an opening brace in an f-string, if
> the format specifier contains "#", the rest of the string is
> considered a comment and the search fails. Therefore, it can be
> worked around by temporarily binding `parse-sexp-ignore-comments' to
> nil, as in the attached patch.
>
> Hangs on gef.py can also be avoided with this patch, but it assumes
> that the braces are properly closed. If you remove the closing braces
> of expressions containing "#", it will eventually hang.
>
[Message part 2 (text/html, inline)]
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Fri, 13 Dec 2024 23:36:02 GMT) Full text and rfc822 format available.Message #23 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: kobarity <kobarity <at> gmail.com> Cc: rehan malak <rehan.malak <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>, 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Fri, 13 Dec 2024 18:35:20 -0500
> The mechanism of the hang is still unclear, but I found the cause of
> the font-lock malfunction: when using `up-list' to find the
> corresponding closing brace for an opening brace in an f-string, if
> the format specifier contains "#", the rest of the string is
> considered a comment and the search fails. Therefore, it can be
> worked around by temporarily binding `parse-sexp-ignore-comments' to
> nil, as in the attached patch.
Maybe a cleaner solution would be to locally use a different
syntax-table (where # doesn't start a comment), but I'm not sure it's
worth the trouble. So I installed your patch into `master`, thank you.
Maybe we should also temporarily setup a narrowing so `up-list` can't
look past `send`.
BTW, beside removing the `font-lock-string-face` from the `face`
property, `python--font-lock-f-strings` should probably also add
a special face for the ":<FMT>" thingies found (optionally) at the end.
> Hangs on gef.py can also be avoided with this patch, but it assumes
> that the braces are properly closed. If you remove the closing braces
> of expressions containing "#", it will eventually hang.
Hmm...
Stefan
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Sun, 15 Dec 2024 14:28:03 GMT) Full text and rfc822 format available.Message #26 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: kobarity <kobarity <at> gmail.com> To: Stefan Monnier <monnier <at> iro.umontreal.ca>, rehan malak <rehan.malak <at> gmail.com> Cc: Eli Zaretskii <eliz <at> gnu.org>, 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Sun, 15 Dec 2024 23:26:05 +0900
[Message part 1 (text/plain, inline)]
rehan malak wrote: > thanks kobarity, with your patch, I confirm that : > - I can open gef.py and scroll > - I get the correct font-lock-string-face in the f-string after a "#" in the format Thank you for your confirmation. Stefan Monnier wrote: > > The mechanism of the hang is still unclear, but I found the cause of > > the font-lock malfunction: when using `up-list' to find the > > corresponding closing brace for an opening brace in an f-string, if > > the format specifier contains "#", the rest of the string is > > considered a comment and the search fails. Therefore, it can be > > worked around by temporarily binding `parse-sexp-ignore-comments' to > > nil, as in the attached patch. > > Maybe a cleaner solution would be to locally use a different > syntax-table (where # doesn't start a comment), but I'm not sure it's > worth the trouble. So I installed your patch into `master`, thank you. Thank you. I agree with you about the syntax-table. > Maybe we should also temporarily setup a narrowing so `up-list` can't > look past `send`. > > BTW, beside removing the `font-lock-string-face` from the `face` > property, `python--font-lock-f-strings` should probably also add > a special face for the ":<FMT>" thingies found (optionally) at the end. > > > Hangs on gef.py can also be avoided with this patch, but it assumes > > that the braces are properly closed. If you remove the closing braces > > of expressions containing "#", it will eventually hang. > > Hmm... Narrowing might improve it, but what about binding `forward-sexp-function' to nil as well, as in the attached patch? It seems to resolve the hang caused by the unbalanced braces in f-strings. 1. wget https://raw.githubusercontent.com/hugsy/gef/refs/heads/main/gef.py 2. sed -i -e 's/\(:#\d*x\)}/\1/g' gef.py 3. emacs -Q gef.py 4. Scroll down the file. I don't think `python-nav-forward-sexp' is necessary here, since the inside of the braces should be an expression, not statements.
[0001-Fix-hangs-caused-by-unbalanced-braces-in-Python-f-st.patch (text/plain, attachment)]
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Sun, 15 Dec 2024 15:04:02 GMT) Full text and rfc822 format available.Message #29 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: kobarity <kobarity <at> gmail.com> Cc: rehan malak <rehan.malak <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>, 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Sun, 15 Dec 2024 10:03:33 -0500
>> > Hangs on gef.py can also be avoided with this patch, but it assumes
>> > that the braces are properly closed. If you remove the closing braces
>> > of expressions containing "#", it will eventually hang.
>>
>> Hmm...
>
> Narrowing might improve it, but what about binding
> `forward-sexp-function' to nil as well, as in the attached patch?
Oh, of course, yes!
Thanks, pushed!
Stefan
bug-gnu-emacs <at> gnu.org:bug#74738; Package emacs.
(Sat, 21 Dec 2024 09:52:02 GMT) Full text and rfc822 format available.Message #32 received at 74738 <at> debbugs.gnu.org (full text, mbox):
From: Eli Zaretskii <eliz <at> gnu.org> To: Stefan Monnier <monnier <at> iro.umontreal.ca> Cc: kobarity <at> gmail.com, rehan.malak <at> gmail.com, 74738 <at> debbugs.gnu.org Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Sat, 21 Dec 2024 11:50:58 +0200
> From: Stefan Monnier <monnier <at> iro.umontreal.ca> > Cc: rehan malak <rehan.malak <at> gmail.com>, Eli Zaretskii <eliz <at> gnu.org>, > 74738 <at> debbugs.gnu.org > Date: Sun, 15 Dec 2024 10:03:33 -0500 > > >> > Hangs on gef.py can also be avoided with this patch, but it assumes > >> > that the braces are properly closed. If you remove the closing braces > >> > of expressions containing "#", it will eventually hang. > >> > >> Hmm... > > > > Narrowing might improve it, but what about binding > > `forward-sexp-function' to nil as well, as in the attached patch? > > Oh, of course, yes! > Thanks, pushed! Thanks, should we now close this bug?
Stefan Monnier <monnier <at> iro.umontreal.ca>:rehan malak <rehan.malak <at> gmail.com>:Message #37 received at 74738-done <at> debbugs.gnu.org (full text, mbox):
From: Stefan Monnier <monnier <at> iro.umontreal.ca> To: Eli Zaretskii <eliz <at> gnu.org> Cc: 74738-done <at> debbugs.gnu.org, kobarity <at> gmail.com, rehan.malak <at> gmail.com Subject: Re: bug#74738: 31.0.50; Freezes in Python-mode on some Python file when searching or scrolling Date: Sat, 21 Dec 2024 09:21:20 -0500
> Thanks, should we now close this bug?
I believe so, yes, closing,
Stefan
Debbugs Internal Request <help-debbugs <at> gnu.org>
to internal_control <at> debbugs.gnu.org.
(Sun, 19 Jan 2025 12:24:08 GMT) Full text and rfc822 format available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.