GNU bug report logs - #27674
26.0.50; cl-progv: strange scoping due to implementation

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27674 in the body.
You can then email your comments to 27674 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 26.0.50; cl-progv: strange scoping due to implementation
Date: Wed, 12 Jul 2017 23:53:01 +0200

Hello,

The way `cl-progv' is implemented, we have some strange effects
happening to closures in the body.  For example, with lexical-binding
on,

(let ((x 0))
  (cl-progv (list 'x) (list 1)
    (funcall (lambda () x))))

yields 0, and

(cl-progv (list 'x) (list 1)
  (funcall (lambda () x)))

yields 1.  That isn't consistent (FWIW I would expect `1' in both
cases).


TIA,

Michael.



In GNU Emacs 26.0.50 (build 7, x86_64-pc-linux-gnu, GTK+ Version 3.22.16)
 of 2017-07-12 built on drachen
Repository revision: dde7f2d48b53996bdf767a8cf91aafc2e10add23
Windowing system distributor 'The X.Org Foundation', version 11.0.11903000
System Description:	Debian GNU/Linux testing (buster)

Message #8 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Wed, 12 Jul 2017 20:21:39 -0400

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> The way `cl-progv' is implemented, we have some strange effects
> happening to closures in the body.  For example, with lexical-binding
> on,
>
> (let ((x 0))
>   (cl-progv (list 'x) (list 1)
>     (funcall (lambda () x))))
>
> yields 0, and
>
> (cl-progv (list 'x) (list 1)
>   (funcall (lambda () x)))
>
> yields 1.  That isn't consistent (FWIW I would expect `1' in both
> cases).

IMO, this is a bug in your program, this yields 1:

(progn
  (defvar x)
  (let ((x 0))
    (cl-progv (list 'x) (list 1)
      (funcall (lambda () x)))))

Note that your second expression gives a compile warning (it's also
missing a defvar, though it happens to give the result you expect even
without that):

27674.el:10:25:Warning: reference to free variable ‘x’

Message #11 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: npostavs <at> users.sourceforge.net
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Thu, 13 Jul 2017 02:36:51 +0200

npostavs <at> users.sourceforge.net writes:

> Michael Heerdegen <michael_heerdegen <at> web.de> writes:
>
> > The way `cl-progv' is implemented, we have some strange effects
> > happening to closures in the body.  For example, with lexical-binding
> > on,
> >
> > (let ((x 0))
> >   (cl-progv (list 'x) (list 1)
> >     (funcall (lambda () x))))
> >
> > yields 0, and
> >
> > (cl-progv (list 'x) (list 1)
> >   (funcall (lambda () x)))
> >
> > yields 1.  That isn't consistent (FWIW I would expect `1' in both
> > cases).
>
> IMO, this is a bug in your program

Why?

>, this yields 1:
>
> (progn
>   (defvar x)
>   (let ((x 0))
>     (cl-progv (list 'x) (list 1)
>       (funcall (lambda () x)))))

Sure, but that's something different.  I didn't want a special variable
in my example.  The doc of `cl-progv' doesn't mention that the symbols
must correspond to special variables.  Do I miss something?


Michael.

Message #14 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Wed, 12 Jul 2017 20:50:35 -0400

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> Sure, but that's something different.  I didn't want a special variable
> in my example.  The doc of `cl-progv' doesn't mention that the symbols
> must correspond to special variables.  Do I miss something?

Oh, you expect cl-progv to bind lexically?  I interpret the first
sentence in its docstring to mean that cl-progv does dynamic binding,
not lexical binding.  Note also, that if it did bind lexically, we would
not be able to compile the body form.

(defmacro cl-progv (symbols values &rest body)
  "Bind SYMBOLS to VALUES dynamically in BODY.
                          ^^^^^^^^^^^

Message #17 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: npostavs <at> users.sourceforge.net
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Thu, 13 Jul 2017 03:11:37 +0200

npostavs <at> users.sourceforge.net writes:

> Michael Heerdegen <michael_heerdegen <at> web.de> writes:
>
> > Sure, but that's something different.  I didn't want a special variable
> > in my example.  The doc of `cl-progv' doesn't mention that the symbols
> > must correspond to special variables.  Do I miss something?
>
> Oh, you expect cl-progv to bind lexically?

Eh - no.  Maybe I have a wrong mental model.  I thought that the free
variable `x' in the lambda is (also) in the scope of the dynamical
binding created by `progv', and because that binding is established
inside the `let' establishing the lexical binding of `x', it would
shadow the lexical binding.

Why does the lambda still refer to the lexical binding?

FWIW I see that this example shows the same behavior:

#+begin_src emacs-lisp
(setq x 'foo)

(let ((x 0))
  (cl-letf (((symbol-value 'x) 1))
    (funcall (lambda () x))))
#+end_src
==> 0

Does a lexical binding always beat a dynamical one?


Michael.

Message #20 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Wed, 12 Jul 2017 21:54:26 -0400

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> Eh - no.  Maybe I have a wrong mental model.  I thought that the free
> variable `x' in the lambda is (also) in the scope of the dynamical
> binding created by `progv', and because that binding is established
> inside the `let' establishing the lexical binding of `x', it would
> shadow the lexical binding.

Oh, you want lexical and dynamic binding on the same variable?  I think
the answer is "don't do that".

> Why does the lambda still refer to the lexical binding?

Maybe it would be more obvious if we wrote it like this:

    (let ((x 0))
      (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
        (funcall (lambda () x))))

Clearly the inner x must refer to the lexical let-binding, right?  Even
if the user happens to enter `x' at the prompt this remains true.

> Does a lexical binding always beat a dynamical one?

Yes, lexical analysis is performed first and then the names are thrown
away, so you can't even tell when the "same" variable has been
dynamically bound as well.

Message #23 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: npostavs <at> users.sourceforge.net
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Thu, 13 Jul 2017 04:15:05 +0200

npostavs <at> users.sourceforge.net writes:

> > Why does the lambda still refer to the lexical binding?
>
> Maybe it would be more obvious if we wrote it like this:
>
>     (let ((x 0))
>       (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
>         (funcall (lambda () x))))
>
> Clearly the inner x must refer to the lexical let-binding, right?  Even
> if the user happens to enter `x' at the prompt this remains true.

Not an argument per se, because with lexical binding mode off, you can
surely do that.

> > Does a lexical binding always beat a dynamical one?
>
> Yes, lexical analysis is performed first and then the names are thrown
> away, so you can't even tell when the "same" variable has been
> dynamically bound as well.

Ok, this is the part I was clearly missing, thanks.  I'll have a look if
the documentation tells something like this (it should be spelled out
somewhere).

Then I guess you can close this report.


Thank you,

Michael.

Message #26 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: npostavs <at> users.sourceforge.net
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Wed, 12 Jul 2017 22:41:12 -0400

tags 27674 notabug wontfix
close 27674
quit

Michael Heerdegen <michael_heerdegen <at> web.de> writes:

> npostavs <at> users.sourceforge.net writes:
>
>> > Why does the lambda still refer to the lexical binding?
>>
>> Maybe it would be more obvious if we wrote it like this:
>>
>>     (let ((x 0))
>>       (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
>>         (funcall (lambda () x))))
>>
>> Clearly the inner x must refer to the lexical let-binding, right?  Even
>> if the user happens to enter `x' at the prompt this remains true.
>
> Not an argument per se, because with lexical binding mode off, you can
> surely do that.

Of course, if `x' is a dynamic variable (e.g., if you use (defvar x) or
you don't have lexical binding enabled) then the inner x refers to the
dynamic binding (again, regardless of what the user enters at the
prompt).

>> > Does a lexical binding always beat a dynamical one?
>>
>> Yes, lexical analysis is performed first and then the names are thrown
>> away, so you can't even tell when the "same" variable has been
>> dynamically bound as well.
>
> Ok, this is the part I was clearly missing, thanks.  I'll have a look if
> the documentation tells something like this (it should be spelled out
> somewhere).

That explanation might be a little bit "infected" by my knowledge of how
the compiler implements lexical binding, the manual carefully talks only
in terms of the "evaluator":

       Here is how lexical binding works.  Each binding construct defines a
    "lexical environment", specifying the variables that are bound within
    the construct and their local values.  When the Lisp evaluator wants
    the current value of a variable, it looks first in the lexical environment. 
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Message #33 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Roland Winkler <winkler <at> gnu.org>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Thu, 13 Jul 2017 09:40:27 -0500

On Wed, Jul 12 2017, npostavs <at> users.sourceforge.net wrote:
>>> > Why does the lambda still refer to the lexical binding?
>>>
>>> Maybe it would be more obvious if we wrote it like this:
>>>
>>>     (let ((x 0))
>>>       (cl-progv (list (intern (read-string "Enter var: "))) (list 1)
>>>         (funcall (lambda () x))))
>>>
>>> Clearly the inner x must refer to the lexical let-binding, right?  Even
>>> if the user happens to enter `x' at the prompt this remains true.
>>
>> Not an argument per se, because with lexical binding mode off, you can
>> surely do that.
>
> Of course, if `x' is a dynamic variable (e.g., if you use (defvar x) or
> you don't have lexical binding enabled) then the inner x refers to the
> dynamic binding (again, regardless of what the user enters at the
> prompt).

I am not sure I can follow this thread:

The docstring of cl-progv says

  Bind SYMBOLS to VALUES dynamically in BODY.

But I am not sure whether this statement correctly reflects the actual
code of cl-progv: cl-progv evaluates a let form at runtime, but it is
not up to cl-progv to ensure dynamical binding.  I believe it depends on
whether lexical binding is on or off whether the code currently used by
cl-progv uses dynamical binding or lexical binding.

So it is my ignorant guess that a consistent behavior of cl-progv with
lexical binding on or off requires that the bindings of SYMBOLS to
VALUES is also passed to `eval' as a second arg.  From the docstring of eval:

  LEXICAL can also be an actual lexical environment, in the form of an
  alist mapping symbols to their value.

Message #36 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> users.sourceforge.net>
To: Roland Winkler <winkler <at> gnu.org>
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Thu, 13 Jul 2017 11:07:47 -0400

On Thu, Jul 13, 2017 at 10:40 AM, Roland Winkler <winkler <at> gnu.org> wrote:
>
> The docstring of cl-progv says
>
>   Bind SYMBOLS to VALUES dynamically in BODY.
>
> But I am not sure whether this statement correctly reflects the actual
> code of cl-progv: cl-progv evaluates a let form at runtime, but it is
> not up to cl-progv to ensure dynamical binding.  I believe it depends on
> whether lexical binding is on or off whether the code currently used by
> cl-progv uses dynamical binding or lexical binding.

No, because cl-progv omits the second argument to `eval', which is the
same as passing nil. This guarantees the evaluated let-bindings are
dynamic bindings.

> So it is my ignorant guess that a consistent behavior of cl-progv with
> lexical binding on or off requires that the bindings of SYMBOLS to
> VALUES is also passed to `eval' as a second arg.

Doing this would mean that cl-progv would always bind lexically. We
could, in theory, change cl-progv to be that way, but it would be
backwards incompatible with previous Emacs versions, and with Common
Lisp.

Message #39 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Noam Postavsky <npostavs <at> users.sourceforge.net>
Cc: 27674 <at> debbugs.gnu.org, Roland Winkler <winkler <at> gnu.org>
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Fri, 14 Jul 2017 16:20:04 +0200

Noam Postavsky <npostavs <at> users.sourceforge.net> writes:

> > The docstring of cl-progv says
> >
> >   Bind SYMBOLS to VALUES dynamically in BODY.
> >
> > But I am not sure whether this statement correctly reflects the actual
> > code of cl-progv: cl-progv evaluates a let form at runtime, but it is
> > not up to cl-progv to ensure dynamical binding.  I believe it
> > depends on
> > whether lexical binding is on or off whether the code currently used by
> > cl-progv uses dynamical binding or lexical binding.
>
> No, because cl-progv omits the second argument to `eval', which is the
> same as passing nil. This guarantees the evaluated let-bindings are
> dynamic bindings.

Yes.  The second part of the answer is the creation of BODYFUN outside
of `eval': it ensures that free variables in the BODY refer to the outer
lexical environment regardless of `eval' being without performed with
lexical binding off.


Michael.

Message #42 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: npostavs <at> users.sourceforge.net
Cc: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Fri, 14 Jul 2017 18:00:57 +0200

npostavs <at> users.sourceforge.net writes:

> That explanation might be a little bit "infected" by my knowledge of how
> the compiler implements lexical binding, the manual carefully talks only
> in terms of the "evaluator":
>
>        Here is how lexical binding works.  Each binding construct defines a
>     "lexical environment", specifying the variables that are bound within
>     the construct and their local values.  When the Lisp evaluator wants
>     the current value of a variable, it looks first in the lexical environment. 
>                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I think that's good enough.


Michael.

Message #45 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: "Roland Winkler" <winkler <at> gnu.org>
To: 27674 <at> debbugs.gnu.org
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Sat, 15 Jul 2017 15:46:32 -0500

For the records:

Essentially, Michael's examples expand to

(let ((x 0))
  (let ((fun (lambda () x)))
    (eval `(let ((x 1))
             (funcall ',fun)))))

(let ((fun (lambda () x)))
  (eval `(let ((x 1))
           (funcall ',fun))))

With dynamic binding, both examples return 1.  With lexical binding,
the compiler complains about

  reference to free variable `x'

in the 1st line of the 2nd example.  Then, the 1st example returns 0,
the 2nd example returns 1.

Message #48 received at 27674 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 27674 <at> debbugs.gnu.org, npostavs <at> users.sourceforge.net
Subject: Re: bug#27674: 26.0.50;
 cl-progv: strange scoping due to implementation
Date: Wed, 19 Jul 2017 09:55:07 -0400

> Ok, this is the part I was clearly missing, thanks.  I'll have a look if
> the documentation tells something like this (it should be spelled out
> somewhere).

FWIW, http://clhs.lisp.se/Body/s_progv.htm gives the following example:

 (let ((*x* 3)) 
    (progv '(*x*) '(4) 
      (list *x* (symbol-value '*x*)))) =>  (3 4)


-- Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.