GNU bug report logs - #50136
28.0.50; A problem with rx-let expansion

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 50136 in the body.
You can then email your comments to 50136 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: bug-gnu-emacs <at> gnu.org
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: 28.0.50; A problem with rx-let expansion
Date: Fri, 20 Aug 2021 15:59:44 +0200

Hello,

I'm referring to this paragraph in rx.el:

;; FIXME: Consider adding extensions in Lisp macro style, where
;; arguments are passed unevaluated to code that returns the rx form
;; to use.
;; [...]
;; While this would permit more powerful extensions, it's unclear just
;; how often they would be used in practice.  Let's wait until there is
;; demand for it.

Ok - here is!  I would find that approach much more natural than the
current one.  Look:

Already the first `rx-let' pseudo macro I tried hit a problem (bug) with
the current approach:

#+begin_src emacs-lisp
(rx-let ((scatter (string)
                  (regex (mapconcat #'string (string-to-list string) ".*"))))
  (rx (scatter "abc"))) => useless error message
#+end_src

while the following version (the only difference is the argument name)
works as expected:

#+begin_src emacs-lisp
(rx-let ((scatter (s) (regex (mapconcat #'string (string-to-list s) ".*"))))
  (rx (scatter "abc"))) ==> "a.*b.*c"
#+end_src

Seems the function form #'string gets replaced and ends as #'"abc" in
the first version because the function name accidentally collides with
the argument name.

Personally I would be more happy with a thing called `rx-macrolet' than
with an extended `rx-let' to support this additional macro-like kind of
syntax.

TIA,

Michael.

Message #8 received at 50136 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: 50136 <at> debbugs.gnu.org
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#50136: 28.0.50; A problem with rx-let expansion
Date: Fri, 20 Aug 2021 16:50:01 +0200

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> #+begin_src emacs-lisp
> (rx-let ((scatter (s) (regex (mapconcat #'string (string-to-list s) ".*"))))
>   (rx (scatter "abc"))) ==> "a.*b.*c"
> #+end_src

AFAIU, the current approach doesn't allow to extend this with an
optional CHARS argument to specify with chars are allowed "in between"
(instead of "any").  The "&optional" keyword makes it barf.

And I can't actually "look" at the arguments to provide a conditional
replacement.  All I can do is to use `regex' to eval a constant
expression I specify (at run-time).  This seems a quite limiting
approach.  Something more macro-like would be appropriate IMO.

Michael.

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Fri, 20 Aug 2021 17:23:58 +0200

20 aug. 2021 kl. 15.59 skrev Michael Heerdegen <michael_heerdegen <at> web.de>:

> (rx-let ((scatter (string)
>                  (regex (mapconcat #'string (string-to-list string) ".*"))))
>  (rx (scatter "abc"))) => useless error message

Yes, that's an unfortunate consequence of the current substitution mechanism (validating Stefan's premonitions to some extent).
However the syndrome is entirely avoidable and does not make the facility less powerful.

> Personally I would be more happy with a thing called `rx-macrolet' than
> with an extended `rx-let' to support this additional macro-like kind of
> syntax.

Noted (in the honest sense)!

> AFAIU, the current approach doesn't allow to extend this with an
> optional CHARS argument to specify with chars are allowed "in between"
> (instead of "any").  The "&optional" keyword makes it barf.

That's right. What would be the preferred semantics? For example, what value would you expect an absent optional parameter to take? `nil` isn't very useful inside rx forms, so it would only make sense inside embedded Lisp expressions.

> And I can't actually "look" at the arguments to provide a conditional
> replacement.  All I can do is to use `regex' to eval a constant
> expression I specify (at run-time).  This seems a quite limiting
> approach.  Something more macro-like would be appropriate IMO.

Actually it's the rx `eval` form that you are looking for if you want arbitrary compile-time computation. `regexp` and `literal` are explicitly made for run-time expressions.

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Fri, 20 Aug 2021 19:21:20 +0200

Mattias Engdegård <mattiase <at> acm.org> writes:

> > AFAIU, the current approach doesn't allow to extend this with an
> > optional CHARS argument to specify with chars are allowed "in between"
> > (instead of "any").  The "&optional" keyword makes it barf.
>
> That's right. What would be the preferred semantics? For example, what
> value would you expect an absent optional parameter to take?

I guess it's not very useful for the current mechanism.  With a
macro-like thing, the expansion would just not use the value if it's nil.  The
expander would check whether the argument was provided.

> Actually it's the rx `eval` form that you are looking for if you want
> arbitrary compile-time computation. `regexp` and `literal` are
> explicitly made for run-time expressions.

Yes, I think you are right.


Michael.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Fri, 20 Aug 2021 20:45:41 +0200

20 aug. 2021 kl. 19.21 skrev Michael Heerdegen <michael_heerdegen <at> web.de>:

> I guess it's not very useful for the current mechanism.  With a
> macro-like thing, the expansion would just not use the value if it's nil.  The
> expander would check whether the argument was provided.

Yes. It was (and is) a trade-off between the simpler "template-like" semantics currently used and the programmatic "execute code" semantics of Lisp macros. The idea was that the simpler semantics would be simpler and sufficient for most uses, and `eval` forms could always be employed in other cases.

I'm still fretting over having made the wrong call, and that at least I should have omitted parametrised rx definitions until better understood.

If you just have one optional argument, try &rest -- at least its default-case semantics are crystal clear (no arguments inserted). Obviously, you could also use separate rx macros for different parameter lists:

(rx-let ((delimited (grp delim) (seq delim (group-n grp (* (not delim)) delim)))
         (double-quoted (grp) (delimited grp ?\")))

etc.

Message #20 received at 50136 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: mattiase <at> acm.org, 50136 <at> debbugs.gnu.org, monnier <at> iro.umontreal.ca,
 rms <at> gnu.org
Subject: Re: bug#50136: 28.0.50; A problem with rx-let expansion
Date: Fri, 20 Aug 2021 23:21:00 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Actually it's the rx `eval` form that you are looking for if you want
  > > arbitrary compile-time computation. `regexp` and `literal` are
  > > explicitly made for run-time expressions.

If there are variables which can hold an rx expression, we need to make
them as unsafe file variable bindings.  Or else have special code
to study them and see if evaluation constructs are used.

Has this been done?

Are these constructs an insecurity?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Sat, 21 Aug 2021 13:45:45 +0200

Mattias Engdegård <mattiase <at> acm.org> writes:

> Yes. It was (and is) a trade-off between the simpler "template-like"
> semantics currently used and the programmatic "execute code" semantics
> of Lisp macros. The idea was that the simpler semantics would be
> simpler and sufficient for most uses, and `eval` forms could always be
> employed in other cases.

Ah ok, now my picture gets somewhat clearer.  The semantics I wish for
are already possible.  I need to use `eval' on the one side, and &rest
on the other.  Since REST from &rest rest is spliced in, I can't use
REST like a variable (and e.g. test whether any args have been provided,
that is impossible, since in that case it expands to nothing); instead I
can use something like

  (let ((args (list rest))) ...)

inside rx `eval' and use that as argument list variable.

I think I can live with that.

Maybe it would be good to improve the docstring to say the things that I
missed clearer? - Say explicitly that this mechanism is different from
macros, instead it's just a simple substitution mechanism, and add a
simple example to the docstring as a reference, even if we already have
examples in the manual.

Maybe also tell that using `eval' you can still have a macro-like
behavior in the end.

Michael.

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Sat, 21 Aug 2021 15:02:47 +0200

21 aug. 2021 kl. 13.45 skrev Michael Heerdegen <michael_heerdegen <at> web.de>:

>  (let ((args (list rest))) ...)
> 
> inside rx `eval' and use that as argument list variable.

Or just put all the actual code in a plain function:

(eval-when-compile
 (defun expand-my-rx-thing (x y &optional z &rest r) ...))

(rx-define my-rx-thing (x y &rest more) (eval (expand-my-rx-thing x y more)))

and use &optional and &rest arguments as you are used to, without any risks of substitution accidents like "string" in the example you showed earlier. The function would effectively work exactly like a macro of the sort you requested.

> Maybe it would be good to improve the docstring to say the things that I
> missed clearer? - Say explicitly that this mechanism is different from
> macros, instead it's just a simple substitution mechanism, and add a
> simple example to the docstring as a reference, even if we already have
> examples in the manual.

I'll see what can be done. It's not really Emacs tradition to have examples in doc strings but maybe they can be improved a bit as you say.

> Maybe also tell that using `eval' you can still have a macro-like
> behavior in the end.

Not a bad idea!

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Mon, 23 Aug 2021 12:45:41 +0200

Mattias Engdegård <mattiase <at> acm.org> writes:

> Or just put all the actual code in a plain function:
>
> (eval-when-compile
>  (defun expand-my-rx-thing (x y &optional z &rest r) ...))
>
> (rx-define my-rx-thing (x y &rest more) (eval (expand-my-rx-thing x y
> more)))
>
> and use &optional and &rest arguments as you are used to, without any
> risks of substitution accidents like "string" in the example you
> showed earlier. The function would effectively work exactly like a
> macro of the sort you requested.

A good idea, I like it!

> > [... talking about docstring tweaks ...]
>
> I'll see what can be done. It's not really Emacs tradition to have
> examples in doc strings but maybe they can be improved a bit as you
> say.

I you find a way to get along without examples, all the better.  I
suggested to use an example because it might be the simplest way to
explain how things work, and because the way things work may come
unexpected for some (like me).  But providing some more details more
explicitly might be as good as well.


Regards,

Michael.

Message #32 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Mon, 23 Aug 2021 14:38:40 +0200

23 aug. 2021 kl. 12.45 skrev Michael Heerdegen <michael_heerdegen <at> web.de>:

>> (rx-define my-rx-thing (x y &rest more) (eval (expand-my-rx-thing x y more)))

> A good idea, I like it!

The variant

 (rx-define my-rx-thing (&rest more) (eval (expand-my-rx-thing more)))

would also work, at the cost of slightly worse error messages if you pass too few arguments to my-rx-thing.

Message #35 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Mon, 23 Aug 2021 17:20:42 +0200

[Message part 1 (text/plain, inline)]

23 aug. 2021 kl. 12.45 skrev Michael Heerdegen <michael_heerdegen <at> web.de>:

> I you find a way to get along without examples, all the better.  I
> suggested to use an example because it might be the simplest way to
> explain how things work, and because the way things work may come
> unexpected for some (like me).  But providing some more details more
> explicitly might be as good as well.

Here is my proposed change to the manual. I didn't change the doc strings since they already refer to that Info node.
Good enough?

[0001-Add-example-of-advanced-user-defined-Rx-form-to-manu.patch (application/octet-stream, attachment)]

Message #38 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Mattias Engdegård <mattiase <at> acm.org>
Cc: bug-gnu-emacs <at> gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Mon, 23 Aug 2021 18:59:11 +0200

Mattias Engdegård <mattiase <at> acm.org> writes:

> Here is my proposed change to the manual. I didn't change the doc
> strings since they already refer to that Info node.

Maybe at least one sentence about that this mechanism just performs
trivial substitution?

> +User-defined forms themselves only perform simple template
> +substitution.

I find the term "user-defined" distracting, because anybody (including
library and package developers) is "affected".  Apart from that, ok for
me.

>  For arbitrary computations, use them together with with
> +the @code{rx} forms @code{eval}, @code{regexp} or @code{literal}.
> +Example:
> +
> +@example
> +@group
> +(defun n-tuple-rx (n element)
> +  `(seq "<"
> +        (group-n 1 ,element)
> +        ,@@(mapcar (lambda (i) `(seq ?, (group-n ,i ,element)))
> +                  (number-sequence 2 n))
> +        ">"))
> +(rx-define n-tuple (n element) (eval (n-tuple-rx n 'element)))
> +(rx (n-tuple 3 (+ (in "0-9"))))
> +  @result{} "<\\(?1:[0-9]+\\),\\(?2:[0-9]+\\),\\(?3:[0-9]+\\)>"
> +@end group
> +@end example
>  @end defmac

Didn't try the example, but it looks good.


I have one more thing, however, once we are here: In the docstring of
`rx', near the end:

| (eval EXPR)    Match the rx sexp from evaluating EXPR at compile time.

can we say "expansion time" instead of "compile time"?


Regards,

Michael.

Message #43 received at 50136-done <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 50136-done <at> debbugs.gnu.org, Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: 28.0.50; A problem with rx-let expansion
Date: Mon, 23 Aug 2021 20:05:26 +0200

23 aug. 2021 kl. 18.59 skrev Michael Heerdegen <michael_heerdegen <at> web.de>:

> Maybe at least one sentence about that this mechanism just performs
> trivial substitution?

I didn't do that now, because I couldn't see how it could be interpreted otherwise (and there's the manual link).

> I find the term "user-defined" distracting, because anybody (including
> library and package developers) is "affected".  Apart from that, ok for
> me.

Thank you, changed.

> Didn't try the example, but it looks good.

Thanks for looking at it. Good examples are always hard to write!

> | (eval EXPR)    Match the rx sexp from evaluating EXPR at compile time.
> 
> can we say "expansion time" instead of "compile time"?

We can, and now do!

Closing; I think we're done here (complain if not).

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.