Subject: bug#72328: [PATCH] Nested backquote in pcase

From: Thuna <[email-masked]>
Date: Fri, 23 Aug 2024 22:19:25 +0200

(Just to address the mail I'm replying to: )
>>> Would be good to hear from Stefan
>> The first step is to declare nested backquotes unsupported and add a
>> warning when we encounter them.
> Where and how would you suggest to do that?

If we say that a leading backquote[1] is unsupported in the backquote
pattern, then we can just add at to the start of the `(consp qpat)'
branch in `pcase--expand-\`' a check like

  (when (eq (car qpat) '\`)
    (warn ...))

and that would do.[2]

I do not really think that making the nested backquotes unsupported is a
good idea - this leaves the only (legal) way to express them in a
backquote pattern as `(,'\` <thing>) which is the thing I am trying to
fix for commas in the first place.  I would rather the original behavior
be documented and marked as intended than to have backquotes be illegal
entirely.

---

Sorry for the radio silence.  I have been a tiny bit occupied and
haven't been able to spend enough time on this.  I still do not really
have any examples I can show, but I have been slowly gathering some
thoughts on this, and I want to put them out there.

>>> Thinking more about it, from the things I can see, I think I'm in favor
>>> of this change.  In the long run it makes Elisp more consistent.

I went in the opposite direction.  The more I think about it, and the
more I try to write examples this would negatively impact, the less
attractive this change starts to look.

>>> I don't know how often backquote values are needed to be matched.  I
>>> guess not terribly often.

This is unfortunately a part of the reason this discussion is not really
going anywhere (with the other part being me, sorry about that!); this
is a *really* niche feature.  I expect that the number of people using
it does not exceed two digits, if even that.

>>> OTOH, this ,'\, circus really can drive none crazy, it's not obvious
>>> what one needs to do at all to match a backquote value correctly, so
>>> it's also an improvement to enable people to do this in a reachable
>>> way, and the code also gets better readable.

Yes, that is what I was hoping for with this change, too, but...

>>> OTOH, I think the current semantics of pcase ` are easier to explain in
>>> a formal way.  Dunno if this is an advantage though, since most people
>>> don't seem to understand them anyway.

...this is really the issue: The old behavior is very easy to wrap your
head around, and its only downside is its inability to express a literal
comma match, from which extends the inability to match against /results/
of a backquote form.

On the other hand, when you are "out of bounds" of the new behavior -
which is whenever you are trying to apply pcase patterns to a
technically quoted form within the pcase[3] - you need to create a
pattern where the normal backquotes go only as deep as your shallowest
comma[4] with the result of the backquotes written using ,'\`.

>>> Thuna, do you have any other ideas where your patch would be a
>>> significant improvement - practical use cases?  I must admit I'm not yet
>>> completely sure how the impact of the patch would on using el-search
>>> - although I often use it to search backquote expressions.

Unfortunately, given the problem above, I would need to discover an
improvement that I don't even know of yet to justify advocating for my
patch.

---

This idea is not yet fleshed out, and it is very complicated one, both
conceptually and implementationally, but it would help maintain
backwards compatibility while allowing a reasonable representation of
literal comma forms in backquote patterns:

What if a ,PAT form would match against PAT as though it were a pcase
pattern /if any only if/ PAT would not error due to a misplaced call to
a comma pattern, and otherwise ,PAT would match against a pcase
backquote subpattern (\, PAT)?

The way it's stated above is not really illustrative, so let me also
explain how it works (as far as everyone will be concerned): Any comma
you put before the actual final comma is matched literally.

Trying to match against an (unevaluated) object of the form

  `(<sybmol> ,<integer>)

would then be done using the pattern

  `(,(pred symbolp) ,,(pred integerp))

and an (evaluated) object like

  ``(let ,(list . <anything>)
      ,,body)

would be matched using the pattern

  ``(let ,(list . ,_)
      ,body)

This comes with its downsides, one of which is that you need to
eventually match /some/ pcase pattern to be able to insert a comma.  For
example, if we had swtiched the object above to

  ``(let ,(list)
      ,,body)

you would not be able to do

  ``(let ,(list)
      ,body)

or

  ``(let ,,(list)
      ,body)

because they would both try to match against `(list)' as though it were
a pcase pattern, but there is a solution, which is to just put a quote
afterwards:

  ``(let ,,'(list)
      ,body)

The actual thing that I have not yet managed to work out is whether this
could cause a significant amount of false positives, that is, silently
matching against a literal comma where an escaping one was intended.  I
am unable to even construct an example, so it may be that it's not
actually possible, but it may also be that I am just not finding it
which I can't discount.

There is some issue with the way this works "backwards", in that you
need to give it more commas the less commas it has (or rather as many
commas as it is apart from the nesting of backquotes - this is difficult
to even explain!), but I think a simplistic interpretation of it as
"give it as many commas as you want to match" would alleviate this
somewhat.

I have a very rough implementation, but it's not actually working
properly and needs a very big overhaul, so I won't (can't) provide it
for others to check out how this would work.  It should be feasible to
do the calculation in your head, so please do that, sorry!

---

[1] Currently, pcase doesn't actually check that a comma form is
"valid", in that it must be a proper list of length 2, but only checks
that the first element in the list is a comma.  As a result, something
like

  (pcase '(1) (`((\, (pred integerp) (error))) t)) ;; => t

"works" even though it should either be a macroexpansion error or
equivalent to the pattern '((\, (pred integerp) (error))).

[2] We could be a bit fancier and have a single pcase form only warn
once with some bookkeeping, but that would still essentially be
something like

  (when (and (eq (car qpat) '\`)
             (not pcase--bad-\`-warnedp))
    (setq pcase--bad-\`-warnedp t)
    (warn ...))

[3] Something like the unevaluated `(<sybmol> ,<integer>) where <symbol>
is quoted as far as the backquote is concerned.

[4] This means that in the example in [3] you have to escape ALL
backquotes with ',\`.