GNU bug report logs -
#66940
Dynamic scoping is all weird now?
Previous Next
Reported by: Dave Goel <deego3 <at> gmail.com>
Date: Sun, 5 Nov 2023 04:08:02 UTC
Severity: normal
Tags: notabug
Done: Michael Heerdegen <michael_heerdegen <at> web.de>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 66940 in the body.
You can then email your comments to 66940 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 04:08:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Dave Goel <deego3 <at> gmail.com>
:
New bug report received and forwarded. Copy sent to
bug-gnu-emacs <at> gnu.org
.
(Sun, 05 Nov 2023 04:08:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I was playing around, checking my sanity re : dynamic scoping. I'm
probably rusty, but I could swear that on older emacsen, this sort of code
would print 33 ten times.
On emacs -Q with debian stable (28.2).
I try this
(progn
(setq lexical-binding nil)
(dotimes (ii 10)
(defmacro mac ()
`(message "%S" ,ii)
(sit-for 0.1))
(let ((old_ii ii))
(setq ii 33)
(mac)
(setq ii old_ii)
)))
Exits without error the first two times, although, strangely, i don't see
any messages the second time. (I expected to see ten 33's).
*(a) As in, the second time, it doesn't print 33 ten times, as we would
expect.*
*(b) The third time is stranger. *
Let's evaluate it a third time! The third time is even weirder.
The third time, it complains
Debugger entered--Lisp error: (void-variable ii)
(list 'message "%S" ii)
(lambda nil (list 'message "%S" ii) (list 'print ii) (sit-for 0.1))()
macroexpand((mac) nil)
macroexp-macroexpand((mac) nil)
macroexp--expand-all((mac))
macroexp--all-forms(((setq ii 33) (mac) (setq ii old_ii)))
macroexp--expand-all((let ((old_ii ii)) (setq ii 33) (mac) (setq ii
old_ii)))
and so on.
The bug(?) was reproduced on 29.1 as well - by e1f on #emacs
*(c) It gets even funnier. e1f changed ii to var, but the third time, it
still complained about ii, with the above error message, and not about var.*
I reproduced that as well.
Dave
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 04:11:01 GMT)
Full text and
rfc822 format available.
Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I apologize.I meant to say it should print 0 to 10 the first time around
because ,ii should hardcode the value at the time of macro definition.
But, the second and third time are so absolutely weird?
On Sun, Nov 5, 2023 at 12:06 AM Dave Goel <deego3 <at> gmail.com> wrote:
> I was playing around, checking my sanity re : dynamic scoping. I'm
> probably rusty, but I could swear that on older emacsen, this sort of code
> would print 33 ten times.
>
>
> On emacs -Q with debian stable (28.2).
> I try this
>
>
> (progn
> (setq lexical-binding nil)
>
> (dotimes (ii 10)
> (defmacro mac ()
> `(message "%S" ,ii)
> (sit-for 0.1))
> (let ((old_ii ii))
> (setq ii 33)
> (mac)
> (setq ii old_ii)
> )))
>
> Exits without error the first two times, although, strangely, i don't see
> any messages the second time. (I expected to see ten 33's).
>
>
> *(a) As in, the second time, it doesn't print 33 ten times, as we would
> expect.*
> *(b) The third time is stranger. *
> Let's evaluate it a third time! The third time is even weirder.
> The third time, it complains
> Debugger entered--Lisp error: (void-variable ii)
> (list 'message "%S" ii)
> (lambda nil (list 'message "%S" ii) (list 'print ii) (sit-for 0.1))()
> macroexpand((mac) nil)
> macroexp-macroexpand((mac) nil)
> macroexp--expand-all((mac))
> macroexp--all-forms(((setq ii 33) (mac) (setq ii old_ii)))
> macroexp--expand-all((let ((old_ii ii)) (setq ii 33) (mac) (setq ii
> old_ii)))
>
> and so on.
>
>
> The bug(?) was reproduced on 29.1 as well - by e1f on #emacs
>
> *(c) It gets even funnier. e1f changed ii to var, but the third time, it
> still complained about ii, with the above error message, and not about var.*
> I reproduced that as well.
>
> Dave
>
>
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 06:06:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 66940 <at> debbugs.gnu.org (full text, mbox):
Dave Goel <deego3 <at> gmail.com> writes:
> On Sun, Nov 5, 2023 at 12:06 AM Dave Goel <deego3 <at> gmail.com> wrote:
>
>> (progn
>> (setq lexical-binding nil)
>>
>> (dotimes (ii 10)
>> (defmacro mac ()
>> `(message "%S" ,ii)
>> (sit-for 0.1))
The macro expansion of mac is what it returns, which is what sit-for
returns...
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 07:00:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 66940 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
got you.
My macro should have been
(defmacro mac()
`(progn
(message "%S" ,ii)
(sit-for 0.1)))
So, the original macro had a brain-fart basically amounted to returning nil
unconditionally.
In that case, why were we getting that strange error?
On Sun, Nov 5, 2023 at 1:04 AM Gerd Möllmann <gerd.moellmann <at> gmail.com>
wrote:
> Dave Goel <deego3 <at> gmail.com> writes:
>
> > On Sun, Nov 5, 2023 at 12:06 AM Dave Goel <deego3 <at> gmail.com> wrote:
> >
> >> (progn
> >> (setq lexical-binding nil)
> >>
> >> (dotimes (ii 10)
> >> (defmacro mac ()
> >> `(message "%S" ,ii)
> >> (sit-for 0.1))
>
> The macro expansion of mac is what it returns, which is what sit-for
> returns...
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 07:27:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 66940 <at> debbugs.gnu.org (full text, mbox):
Dave Goel <deego3 <at> gmail.com> writes:
> (defmacro mac()
> `(progn
> (message "%S" ,ii)
> (sit-for 0.1)))
>
> So, the original macro had a brain-fart basically amounted to returning nil
> unconditionally.
> In that case, why were we getting that strange error?
Don't know exactly what the "strage error" refers to, but maybe a (pp
(macrexpand-all ...)) helps? In this case, the result is
"(progn
(setq lexical-binding nil)
(let ((upper-bound 10) (counter 0))
(while (< counter upper-bound)
(let ((ii counter))
(defalias 'mac
(cons 'macro
#'(lambda nil (list 'message \"%S\" ii) (sit-for 0.1))))
(let ((old_ii ii)) (setq ii 33) (mac) (setq ii old_ii)))
(setq counter (1+ counter)))))
One things that gets kind of more obvious is that defmacro is something
"global" in the sense that it sets mac's symbol function. You could
moved the defmacro out of the progn somewhere else.
Does that help?
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 08:13:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 66940 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Sun, Nov 5, 2023 at 2:25 AM Gerd Möllmann <gerd.moellmann <at> gmail.com>
wrote:
>
> Don't know exactly what the "strage error" refers to
>
My original code may not make logical sense, but it was valid code?
It shouldn't have produced the errors described in sections (b) and (c) of
the bug report?
Those are both emacs bugs in (b) and (c)?
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 08:41:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 66940 <at> debbugs.gnu.org (full text, mbox):
Dave Goel <deego3 <at> gmail.com> writes:
>>
>> Don't know exactly what the "strage error" refers to
>>
>
> My original code may not make logical sense, but it was valid code?
>
> It shouldn't have produced the errors described in sections (b) and (c) of
> the bug report?
>
> Those are both emacs bugs in (b) and (c)?
I must admit that I couldn't follow (c). In case (b) is don't see a bug.
Maybe you could write some easier to understand code which demonstates
what you mean?
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 19:08:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 66940 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Gerd,
Ok.
I hope this bugreport is clearer -
Here's some code.
The variable ii is clearly set every time mac was defined OR called.
backquote should expand, and mac should have read its content as (list
'message "%S" ii).
(progn
(setq lexical-binding nil)
(dotimes (ii 10)
(defmacro mac ()
`(message "%S" ,ii)
)
(let
((old_ii ii))
(setq ii 33)
(mac)
(setq ii old_ii)
)))
You eval this code once. It works.
You eval this again. It works.
The third time, though, it lands you in the debugger. The very same code.
Why the third time? And, why the debugger? ii is well set every time it is
used.
That was bug (b). That's clearly an emacs bug?
Here's (c) -
The third time, change every "ii" to "var" ,and change every "old_ii" to
"old_var".
And, evaluate it.
We still get the very same error, but the error involves ii, not var. The
code we are defining and eval'ing has no "ii" whatsoever. It should have
forgotten the old mac that involved ii. We defined a new mac involving
"var."
>
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Sun, 05 Nov 2023 20:23:01 GMT)
Full text and
rfc822 format available.
Message #29 received at 66940 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Here's the simplest MRE I have so far -
(progn
(setq lexical-binding nil)
(let ((ii 1))
(defmacro mac ()
`(message "%S" ,ii)
)
(mac)))
Eval it three times, using C-x C-e.
Error on the third time.
(Following the prior response, I figured I will discuss on devel first to
confirm before officially reporting it, but was kinda directed back here.)
On Sun, Nov 5, 2023 at 2:07 PM Dave Goel <deego3 <at> gmail.com> wrote:
> Gerd,
>
> Ok.
>
> I hope this bugreport is clearer -
>
> Here's some code.
>
> The variable ii is clearly set every time mac was defined OR called.
>
> backquote should expand, and mac should have read its content as (list
> 'message "%S" ii).
>
>
>
>
> (progn
> (setq lexical-binding nil)
>
> (dotimes (ii 10)
> (defmacro mac ()
> `(message "%S" ,ii)
> )
> (let
> ((old_ii ii))
> (setq ii 33)
> (mac)
> (setq ii old_ii)
> )))
>
>
> You eval this code once. It works.
> You eval this again. It works.
> The third time, though, it lands you in the debugger. The very same code.
> Why the third time? And, why the debugger? ii is well set every time it is
> used.
>
> That was bug (b). That's clearly an emacs bug?
>
>
> Here's (c) -
>
> The third time, change every "ii" to "var" ,and change every "old_ii" to
> "old_var".
> And, evaluate it.
> We still get the very same error, but the error involves ii, not var. The
> code we are defining and eval'ing has no "ii" whatsoever. It should have
> forgotten the old mac that involved ii. We defined a new mac involving
> "var."
>
>
>>
>>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Mon, 06 Nov 2023 01:53:01 GMT)
Full text and
rfc822 format available.
Message #32 received at 66940 <at> debbugs.gnu.org (full text, mbox):
Dave Goel <deego3 <at> gmail.com> writes:
> (progn
> (setq lexical-binding nil)
>
> (dotimes (ii 10)
> (defmacro mac ()
> `(message "%S" ,ii)
> )
> (let
> ((old_ii ii))
> (setq ii 33)
> (mac)
> (setq ii old_ii)
> )))
This code has two problems:
(1) Setting `lexical-binding' in the middle of an evaluation does not
work as you think. Either you eval en expression (like the above) using
lexical binding or dynamical binding. If you change `lexical-binding'
in the middle you might get unexpected behavior. Please set the binding
mode only in the file header, not in the code. In the extremely rare
cases where you really must evaluate an expression using the other
binding mode use `eval' with an appropriate second argument.
(2) Macro expansion does not work as you think. Most of the time macros
are expanded _once_ in the complete expression and then the result is
evaluated. When you redefine a macro in the middle of evaluating code
using it, most of the time this will not have an effect because the
macro had already been expanded in the following code.
> You eval this code once. It works.
> You eval this again. It works.
> The third time, though, it lands you in the debugger. The very same
> code. Why the third time? And, why the debugger? ii is well set every
> time it is used.
It's not surprising.
The first time you start evaluating the expression using lexical
binding. Because your code sets lexical-binding to nil, the second time
you actually use dynamical binding completely.
The third run fails because when `mac' is expanded a variable `ii' is
undefined on top-level, so macroexpansion fails.
The second run is different: because the first run had used
lexical-binding (more or less completely, since you started the
interpreter using lexical-binding mode), the macroexpander of `mac' is
actually a closure that inherited the value of `ii' from the last
iteration of the first run. Because of that the second run succeeds.
The second run redefines `mac' so that its reference to `ii' now means
the dynamical global variable, which doesn't exist.
So, I think everything indeed perfectly works as expected here.
Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Mon, 06 Nov 2023 06:27:02 GMT)
Full text and
rfc822 format available.
Message #35 received at 66940 <at> debbugs.gnu.org (full text, mbox):
Michael Heerdegen <michael_heerdegen <at> web.de> writes:
> So, I think everything indeed perfectly works as expected here.
Thanks for explaining this much better than I could have done it.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Mon, 06 Nov 2023 06:59:01 GMT)
Full text and
rfc822 format available.
Message #38 received at 66940 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Michael,
Many thanks for that excellent explanation.
One more question -
With this, first, let's simplify our example as follows.
Let's eval this at the start:
(setq lexical-binding nil)
Eval it, so we are operating in the old world of dynamic scoping.
(let ((ii 1))
(defmacro mac ()
`(print ,ii)
)
(mac))
As you said, with lexical-binding nil, the ii = 1 had no effect above. It
will use the ii from the runtime environment.
So, let's try this -
(setq ii 3)
(let ((ii 4))
(mac))
It evals to 3, not 4. Per our understanding of dynamic scoping, we would
have expected it use the innermost ii? Why does it use the global ii?
On Sun, Nov 5, 2023 at 8:52 PM Michael Heerdegen <michael_heerdegen <at> web.de>
wrote:
> Dave Goel <deego3 <at> gmail.com> writes:
>
> > (progn
> > (setq lexical-binding nil)
> >
> > (dotimes (ii 10)
> > (defmacro mac ()
> > `(message "%S" ,ii)
> > )
> > (let
> > ((old_ii ii))
> > (setq ii 33)
> > (mac)
> > (setq ii old_ii)
> > )))
>
> This code has two problems:
>
> (1) Setting `lexical-binding' in the middle of an evaluation does not
> work as you think. Either you eval en expression (like the above) using
> lexical binding or dynamical binding. If you change `lexical-binding'
> in the middle you might get unexpected behavior. Please set the binding
> mode only in the file header, not in the code. In the extremely rare
> cases where you really must evaluate an expression using the other
> binding mode use `eval' with an appropriate second argument.
>
> (2) Macro expansion does not work as you think. Most of the time macros
> are expanded _once_ in the complete expression and then the result is
> evaluated. When you redefine a macro in the middle of evaluating code
> using it, most of the time this will not have an effect because the
> macro had already been expanded in the following code.
>
>
> > You eval this code once. It works.
> > You eval this again. It works.
> > The third time, though, it lands you in the debugger. The very same
> > code. Why the third time? And, why the debugger? ii is well set every
> > time it is used.
>
> It's not surprising.
>
> The first time you start evaluating the expression using lexical
> binding. Because your code sets lexical-binding to nil, the second time
> you actually use dynamical binding completely.
>
> The third run fails because when `mac' is expanded a variable `ii' is
> undefined on top-level, so macroexpansion fails.
>
> The second run is different: because the first run had used
> lexical-binding (more or less completely, since you started the
> interpreter using lexical-binding mode), the macroexpander of `mac' is
> actually a closure that inherited the value of `ii' from the last
> iteration of the first run. Because of that the second run succeeds.
>
>
> The second run redefines `mac' so that its reference to `ii' now means
> the dynamical global variable, which doesn't exist.
>
> So, I think everything indeed perfectly works as expected here.
>
>
> Michael.
>
[Message part 2 (text/html, inline)]
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Mon, 06 Nov 2023 16:50:02 GMT)
Full text and
rfc822 format available.
Message #41 received at 66940 <at> debbugs.gnu.org (full text, mbox):
> > So, I think everything indeed perfectly works as expected here.
>
> Thanks for explaining this much better than I could have done it.
Ditto. And for helping me _understand_ it.
Added tag(s) notabug.
Request was from
Michael Heerdegen <michael_heerdegen <at> web.de>
to
control <at> debbugs.gnu.org
.
(Tue, 07 Nov 2023 05:45:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Tue, 07 Nov 2023 06:01:02 GMT)
Full text and
rfc822 format available.
Message #46 received at 66940 <at> debbugs.gnu.org (full text, mbox):
Dave Goel <deego3 <at> gmail.com> writes:
> Let's eval this at the start:
>
> (setq lexical-binding nil)
>
> Eval it, so we are operating in the old world of dynamic scoping.
Ok, let's do this.
> (let ((ii 1))
> (defmacro mac ()
> `(print ,ii)
> )
> (mac))
So now:
(symbol-function 'mac)
==> (macro . (lambda () (list 'print ii)))
> [...]
> So, let's try this -
>
> (setq ii 3)
> (let ((ii 4))
> (mac))
>
> It evals to 3, not 4. Per our understanding of dynamic scoping, we
> would have expected it use the innermost ii? Why does it use the
> global ii?
Remember that macros are not expanded on the fly - they are expanded
once for the complete expression in a separate step _before_ the actual
evaluation. So:
(setq ii 3)
; ii --> 3
Expansion of the second expression:
(let ((ii 4))
(mac))
~~> (let ((ii 4)) (print 3))
^^^^^^^^^
expansion of (mac)
Evaluation:
(let ((ii 4)) (print 3))
; prints "3"
A different thing would be to define
(defmacro mac ()
'(print ii))
Then the expansion would still reference the variable. But you could as
well just write (print ii) then.
To sum up: you are using macros wrong in this case. Macros are not
expanded at run-time. It is important to remember that.
Note that not even `cl-macrolet' works like this: the definition and
scope of the defined macros is local, but they are still expanded before
evaluation, so the expansion can't refer to run-time bindings as well.
Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Wed, 08 Nov 2023 02:51:02 GMT)
Full text and
rfc822 format available.
Message #49 received at 66940 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Thank you
[Message part 2 (text/html, inline)]
Reply sent
to
Michael Heerdegen <michael_heerdegen <at> web.de>
:
You have taken responsibility.
(Wed, 08 Nov 2023 03:09:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Dave Goel <deego3 <at> gmail.com>
:
bug acknowledged by developer.
(Wed, 08 Nov 2023 03:09:02 GMT)
Full text and
rfc822 format available.
Message #54 received at 66940-done <at> debbugs.gnu.org (full text, mbox):
Dave Goel <deego3 <at> gmail.com> writes:
> Thank you
Then let's close this one.
What I still wanted to say (to you): no need to complicate things using
macros here in your example. Use functions, then variable references
are simple and clear.
If a macro (expansion) needs to refer to run-time values, use macro
arguments to pass an appropriate expression (like a variable).
But this is overkill in your case, no macros needed. The purpose of
Lisp macros is to extend the language and/or rewrite code (at compile
time). More or less, only that. Always think twice if you really need
macros, try to avoid using macros when possible. This will make the
code easier to understand and maintain. Only if a macro introduces an
abstraction that makes the code actually easier to understand justifies
using one.
Michael.
Information forwarded
to
bug-gnu-emacs <at> gnu.org
:
bug#66940
; Package
emacs
.
(Wed, 08 Nov 2023 03:23:02 GMT)
Full text and
rfc822 format available.
Message #57 received at 66940-done <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Most absolutely, and of course.
I was just trying macros here to improve my understanding.
Which it happened, thanks to all y'all's excellent explanations.
On Tue, Nov 7, 2023 at 10:07 PM Michael Heerdegen <michael_heerdegen <at> web.de>
wrote:
> Dave Goel <deego3 <at> gmail.com> writes:
>
> > Thank you
>
> Then let's close this one.
>
> What I still wanted to say (to you): no need to complicate things using
> macros here in your example. Use functions, then variable references
> are simple and clear.
>
> If a macro (expansion) needs to refer to run-time values, use macro
> arguments to pass an appropriate expression (like a variable).
>
> But this is overkill in your case, no macros needed. The purpose of
> Lisp macros is to extend the language and/or rewrite code (at compile
> time). More or less, only that. Always think twice if you really need
> macros, try to avoid using macros when possible. This will make the
> code easier to understand and maintain. Only if a macro introduces an
> abstraction that makes the code actually easier to understand justifies
> using one.
>
> Michael.
>
[Message part 2 (text/html, inline)]
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Wed, 06 Dec 2023 12:24:06 GMT)
Full text and
rfc822 format available.
This bug report was last modified 1 year and 198 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.