Subject: bug#72478: 29.2.50; down-list isn't customizable by modes like backward-up-list

From: Spencer Baugh <[email-masked]>
Date: Mon, 05 Aug 2024 11:28:48 -0400

The docstrings of both backward-up-list (C-M-u) and down-list (C-M-d)
say:

"This command will also work on other parentheses-like expressions
defined by the current language mode."

However, this is misleading:
- backward-up-list (and up-list) will use forward-sexp-function
  if that variable is non-nil, and otherwise use scan-lists
- down-list does not check forward-sexp-function and always uses
  scan-lists.

So, the two commands can have significantly different behavior in modes
that configure forward-sexp-function, in particular in modes which give
forward-sexp-function awareness of language-specific syntax which
scan-lists can't handle.

For example, python-mode sets forward-sexp-function to a function which
can operate on both lists and Python syntax expressions.  This results
in backward-up-list and down-list having completely different behavior
from each other.

I suggest we should take one of these options:

A. Update the docstring of up-list/down-list to indicate that while
   they're customizable by the mode, they aren't likely to work the same
   if they are customized by the mode.

B. Add a down-list-function buffer-local variable so that modes which
   are already configuring forward-sexp-function can also configure
   down-list.

C. Add a scan-expressions-function buffer-local variable, and have both
   backward-up-list and down-list check this variable.  Again, modes
   which are already configuring forward-sexp-function to work with
   language-specific syntax can configure this variable.

I personally suggests C.  My major-mode currently implements
forward-sexp-function, and it would be easy for it to also set
scan-expressions-function.  This would also fit well with
tree-sitter-based modes: a tree-sitter based mode could easily have a
tree-sitter-based scan-expressions-function.

I'm interested in implementing this scan-expressions-function variable,
if that sounds like a reasonable solution.

In GNU Emacs 29.2.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.15.12, Xaw scroll bars) of 2024-08-04 built on
 igm-qws-hydra336
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Rocky Linux 8.10 (Green Obsidian)

Configured using:
 'configure --config-cache --with-x-toolkit=lucid --without-gpm
 --without-gconf --without-selinux --without-imagemagick --with-modules
 --with-gif=no --with-cairo --with-rsvg --without-compress-install
 --with-tree-sitter --prefix=/j/office/app/emacs/store/20240804-224533
 PKG_CONFIG_PATH=/usr/local/home/garnish/libtree-sitter/0.22.6-1/lib/pkgconfig/'

Configured features:
CAIRO DBUS FREETYPE GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND
SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental