GNU bug report logs - #69533
30.0.50; Wrong byte compilation of a certain apply syntax

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 69533 in the body.
You can then email your comments to 69533 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.50; Wrong byte compilation of a certain apply syntax
Date: Mon, 04 Mar 2024 02:51:42 +0100

Hello,

C-h f apply says:

| ...
| With a single argument, call the argument's first element using the
| other elements as args.

Issue 1:  This doc sentence has to be moved after the following example,
because that example:

| Thus, (apply '+ 1 2 '(3 4)) returns 10.

is an example for the more widespread syntax.

That sentence actually explains a special case: it tells that this is
also allowed:

  (apply '(+ 1 2)) ==> 3


Issue 2: The byte compiler currently miscompiles such expressions:

Expected:

  (funcall (lambda () (apply '(+ 1 2)))) ==> 3

but

  (funcall (byte-compile '(lambda () (apply '(+ 1 2)))))
   ~~> Error: Invalid function: (+ 1 2)

AFAIU this is `byte-optimize-apply's fault:

(byte-optimize-apply '(apply '(+ 1 2)))
 ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!


TIA,

Michael.

Message #8 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Michael Heerdegen <michael_heerdegen <at> web.de>, "69533 <at> debbugs.gnu.org"
 <69533 <at> debbugs.gnu.org>
Cc: "rms <at> gnu.org" <rms <at> gnu.org>
Subject: RE: [External] : bug#69533: 30.0.50; Wrong byte compilation of a
 certain apply syntax
Date: Mon, 4 Mar 2024 03:39:46 +0000

> | With a single argument, call the argument's
> | first element using the other elements as args.

> ...
> That sentence actually explains a special case:
> it tells that this is also allowed:
>   (apply '(+ 1 2)) ==> 3

FWIW, I'm very surprised to find this "feature".
That it's documented suggests it was intended.
But I don't know why it would be.  Does anyone?
What's the use case for not requiring the first
arg to be a function?

No Lisp that I'm aware of, starting with Lisp 1,
has ever supported such a feature - at least not
according to its doc.

Ccing RMS, who might know why, if this has been
in Elisp since Day One or it was added long ago.
___

Aha, I see now - found it.  It was added in Emacs
28.1.

And this text was added then to the Elisp manual:

  ‘apply’ with a single argument is special:
  the first element of the argument, which
  must be a non-empty list, is called as a
  function with the remaining elements as
  individual arguments.  Passing two or more
  arguments will be faster.

And this additional example was added there:

    (apply '(+ 3 4))
         ⇒ 7

But why?  There's nothing in NEWS about this.
Was it even discussed? (Probably, but maybe
only in a bug thread?)

None of the rest of the text is changed -
just what I've mentioned here.

I don't understand why anyone considered it
a good idea to change the behavior of `apply'
(in this way or in any other way).  What
gives?

"Special" is indeed one way to characterize
it.  More like "shocking". ;-)  What's next?

And this bug is about the behavior not even
faithfully reflecting that doc?  Sheesh.

Was this done on April 1?

Message #11 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Drew Adams <drew.adams <at> oracle.com>
Cc: "69533 <at> debbugs.gnu.org" <69533 <at> debbugs.gnu.org>,
 "rms <at> gnu.org" <rms <at> gnu.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 05:40:36 +0100

Drew Adams <drew.adams <at> oracle.com> writes:

> FWIW, I'm very surprised to find this "feature".
> That it's documented suggests it was intended.
> But I don't know why it would be.  Does anyone?

Bug#40968 I think.

Michael.

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Michael Heerdegen via "Bug reports for GNU Emacs, the Swiss army knife
 of text editors" <bug-gnu-emacs <at> gnu.org>
Cc: 69533 <at> debbugs.gnu.org
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 06:13:05 +0100

Michael Heerdegen via "Bug reports for GNU Emacs, the Swiss army knife
of text editors" <bug-gnu-emacs <at> gnu.org> writes:

> (byte-optimize-apply '(apply '(+ 1 2)))
>  ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!

I'm adding Mattias to the discussion - he had changed
`byte-optimize-apply' after this new convention (which I find
irritating, too) had been introduced.

I guess since byte compilation of such forms is (at least partly)
broken, not many people are actually using it.

Michael.

Message #20 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <basil <at> contovou.net>
To: 69533 <at> debbugs.gnu.org
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 08:46:54 +0100

Michael Heerdegen [2024-03-04 06:13 +0100] wrote:

> Michael Heerdegen <bug-gnu-emacs <at> gnu.org> writes:
>
>> (byte-optimize-apply '(apply '(+ 1 2)))
>>  ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!
>
> I'm adding Mattias to the discussion - he had changed
> `byte-optimize-apply' after this new convention (which I find
> irritating, too) had been introduced.

Just curious: which convention are you referring to, and why do you say
it's new?  AFAICT the only recent Emacs version which accepted
(apply '(+ 1 2)) without any complaint is Emacs 28.

> I guess since byte compilation of such forms is (at least partly)
> broken, not many people are actually using it.

FWIW, my guilty pleasure is using this calling convention with
a non-literal form, e.g.:

  (define-advice foo (:around (&rest args) my-foo)
    (with-something (apply args)))

This has always byte-compiled without issue.
Of course, I'm not married to this convention.

Thanks,
-- 
Basil

Message #23 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: "Basil L. Contovounesios" <basil <at> contovou.net>
Cc: 69533 <at> debbugs.gnu.org,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 09:10:42 +0100

"Basil L. Contovounesios" <basil <at> contovou.net> writes:

> Just curious: which convention are you referring to, and why do you
> say it's new?

The convention that (apply (F . ARGS)) == (apply F . ARGS).

>  AFAICT the only recent Emacs version which accepted
> (apply '(+ 1 2)) without any complaint is Emacs 28.

So it is quite new.

> FWIW, my guilty pleasure is using this calling convention with
> a non-literal form, e.g.:
>
>   (define-advice foo (:around (&rest args) my-foo)
>     (with-something (apply args)))
>
> This has always byte-compiled without issue.

Aha - a user!

No, `byte-optimize-apply' does not mess up this case:

  (byte-optimize-apply '(apply args)) --> (apply args)

so it will compile just fine but

  (byte-optimize-apply '(apply '(+ 1 2)))
   --> (funcall '(+ 1 2) '+ '1 '2) ; broken

That the byte compiler result is (sometimes) broken is a bug in the
optimization code, not in the compiler itself.

Michael.

Message #26 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: "Basil L. Contovounesios" <basil <at> contovou.net>
Cc: 69533 <at> debbugs.gnu.org,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 09:17:20 +0100

"Basil L. Contovounesios" <basil <at> contovou.net> writes:

> FWIW, my guilty pleasure is using this calling convention with
> a non-literal form, e.g.:
>
>   (define-advice foo (:around (&rest args) my-foo)
>     (with-something (apply args)))

Hmm, I realize now that I have used this convention all the time, too,
without thinking about it (it just worked...), in the same
situation.  Instead of - what? - (apply #'funcall args)?  But I would
not say I like it.

Michael.

Message #29 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <basil <at> contovou.net>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 69533 <at> debbugs.gnu.org,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 09:47:11 +0100

Michael Heerdegen [2024-03-04 09:10 +0100] wrote:

> "Basil L. Contovounesios" <basil <at> contovou.net> writes:
>
>> Just curious: which convention are you referring to, and why do you
>> say it's new?
>
> The convention that (apply (F . ARGS)) == (apply F . ARGS).
>
>>  AFAICT the only recent Emacs version which accepted
>> (apply '(+ 1 2)) without any complaint is Emacs 28.
>
> So it is quite new.

Depends: in interpreted code, it's worked since at least as far back as
Emacs 24.  Emacs 28 is the only version in which it successfully
byte-compiles, and doesn't emit any warnings.  Emacs 29 and 30 choke.

-- 
Basil

Message #32 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <basil <at> contovou.net>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: 69533 <at> debbugs.gnu.org,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 09:48:31 +0100

Michael Heerdegen [2024-03-04 09:17 +0100] wrote:

> "Basil L. Contovounesios" <basil <at> contovou.net> writes:
>
>> FWIW, my guilty pleasure is using this calling convention with
>> a non-literal form, e.g.:
>>
>>   (define-advice foo (:around (&rest args) my-foo)
>>     (with-something (apply args)))
>
> Hmm, I realize now that I have used this convention all the time, too,
> without thinking about it (it just worked...), in the same
> situation.  Instead of - what? - (apply #'funcall args)?

No, instead of (apply (car args) (cdr args)), or

  (define-advice foo (:around (fn &rest args) my-foo)
    (with-something (apply fn args)))

-- 
Basil

Message #35 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> suse.de>
To: "Basil L. Contovounesios" <basil <at> contovou.net>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>, 69533 <at> debbugs.gnu.org,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 10:39:51 +0100

On Mär 04 2024, Basil L. Contovounesios wrote:

> Just curious: which convention are you referring to, and why do you say
> it's new?  AFAICT the only recent Emacs version which accepted
> (apply '(+ 1 2)) without any complaint is Emacs 28.

It was added in commit 8edd4a2b64e, since Emacs 24.

-- 
Andreas Schwab, SUSE Labs, schwab <at> suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Message #38 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: "Basil L. Contovounesios" <basil <at> contovou.net>
To: Andreas Schwab <schwab <at> suse.de>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>, 69533 <at> debbugs.gnu.org,
 Mattias Engdegård <mattiase <at> acm.org>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 04 Mar 2024 13:52:55 +0100

Andreas Schwab [2024-03-04 10:39 +0100] wrote:

> On Mär 04 2024, Basil L. Contovounesios wrote:
>
>> Just curious: which convention are you referring to, and why do you say
>> it's new?  AFAICT the only recent Emacs version which accepted
>> (apply '(+ 1 2)) without any complaint is Emacs 28.
>
> It was added in commit 8edd4a2b64e, since Emacs 24.

Right, but that doesn't mean literal forms survive byte-compilation:

  $ cd "$(mktemp -d)"
  $ cat << EOF > foo.el
  > ; -*- lexical-binding: t -*-
  > (message "%s" (apply '(1+ 0)))
  > EOF
  $ emacs-24.5 -Q -batch -f batch-byte-compile foo.el

  In toplevel form:
  foo.el:2:1:Warning: `(1+ 0)' is a malformed function
  Wrote /tmp/tmp.aFbeBGQj7q/foo.elc
  $ emacs-24.5 -Q -script foo.elc
  Invalid function: (1+ 0)

By contrast:

  $ emacs-28.2 -Q -batch -f batch-byte-compile foo.el
  $ emacs-28.2 -Q -script foo.elc
  1

-- 
Basil

Message #41 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: "Basil L. Contovounesios" <basil <at> contovou.net>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>,
 Andreas Schwab <schwab <at> suse.de>, Stefan Monnier <monnier <at> iro.umontreal.ca>,
 69533 <at> debbugs.gnu.org
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 4 Mar 2024 14:13:00 +0100

This was probably just a mistake of mine; a `<=` should have been `<`. Sorry about that, I'll fix.

I think Stefan introduced the 1-arg `apply` semantics as a principled and natural extension to the 2-or-more-arg case, but I'm less convinced. In any case, it's there and documented.

However it is tempting to see the spotty implementation and bugs in many versions as the converse of Hyrum's law that lets us remove the feature instead...

Message #46 received at 69533-done <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: "Basil L. Contovounesios" <basil <at> contovou.net>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>,
 Andreas Schwab <schwab <at> suse.de>, 69533-done <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Mon, 4 Mar 2024 14:25:50 +0100

> This was probably just a mistake of mine; a `<=` should have been `<`.

Actually `>=` should have been `>`.
Pushed to master; closing the bug.

Message #49 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: "69533 <at> debbugs.gnu.org" <69533 <at> debbugs.gnu.org>,
 "rms <at> gnu.org" <rms <at> gnu.org>
Subject: RE: [External] : Re: bug#69533: 30.0.50; Wrong byte compilation of a
 certain apply syntax
Date: Mon, 4 Mar 2024 15:52:20 +0000

> Drew Adams writes:
> 
> > FWIW, I'm very surprised to find this "feature".
> > That it's documented suggests it was intended.
> > But I don't know why it would be.  Does anyone?
> 
> Bug#40968 I think.

Yes, thanks.  I'd forgotten about that,
and that I'd contributed to the thread.

I agree with what I said there ;-),
which is not always the case. ;-)

In particular this:

  Pip>> I think there's consensus, then. 

  me> FTR, FWIW: Not a consensus that includes me.

  me> I'm in favor of making an incompatible
  me> change, to align Emacs with Common Lisp's
  me> more reasonable behavior.

I might have added every other Lisp; it's
not just Common Lisp, AFAIK.

No one has given a good reason _why_ Elisp
ever had this one-off (no other Lisp has
it) behavior, AFAICS.

There's no reason given in that thread.
The thread just concentrates on "fixing"
the odd-ball "special" case.  And the fix
was to mention that case in the doc.

Really too bad.

Message #52 received at 69533-done <at> debbugs.gnu.org (full text, mbox):

From: Drew Adams <drew.adams <at> oracle.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>, "Basil
 L. Contovounesios" <basil <at> contovou.net>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>,
 Andreas Schwab <schwab <at> suse.de>,
 "69533-done <at> debbugs.gnu.org" <69533-done <at> debbugs.gnu.org>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: RE: [External] : bug#69533: 30.0.50; Wrong byte compilation of a
 certain apply syntax
Date: Mon, 4 Mar 2024 21:12:56 +0000

> Pushed to master; closing the bug.

The real bug was the misguided addition of
this "feature" to Elisp, making it unlike
all other Lisps (AFAIK).

How about we fix that instead?

Message #55 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>,
 Mattias Engdegård <mattiase <at> acm.org>
Cc: 69533 <at> debbugs.gnu.org
Subject: Re: bug#69533: 30.0.50;
 Wrong byte compilation of a certain apply syntax
Date: Thu, 14 Mar 2024 09:52:52 +0200

> Date: Mon, 04 Mar 2024 02:51:42 +0100
> From:  Michael Heerdegen via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs <at> gnu.org>
> 
> 
> Hello,
> 
> C-h f apply says:
> 
> | ...
> | With a single argument, call the argument's first element using the
> | other elements as args.
> 
> Issue 1:  This doc sentence has to be moved after the following example,
> because that example:
> 
> | Thus, (apply '+ 1 2 '(3 4)) returns 10.
> 
> is an example for the more widespread syntax.
> 
> That sentence actually explains a special case: it tells that this is
> also allowed:
> 
>   (apply '(+ 1 2)) ==> 3
> 
> 
> Issue 2: The byte compiler currently miscompiles such expressions:
> 
> Expected:
> 
>   (funcall (lambda () (apply '(+ 1 2)))) ==> 3
> 
> but
> 
>   (funcall (byte-compile '(lambda () (apply '(+ 1 2)))))
>    ~~> Error: Invalid function: (+ 1 2)
> 
> AFAIU this is `byte-optimize-apply's fault:
> 
> (byte-optimize-apply '(apply '(+ 1 2)))
>  ==> (funcall '(+ 1 2) '+ '1 '2) ; Ouch!
> 

Stefan and Mattias, any comments or suggestions?

Message #58 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Michael Heerdegen <michael_heerdegen <at> web.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>, 69533 <at> debbugs.gnu.org
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Thu, 14 Mar 2024 09:44:26 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

> Stefan and Mattias, any comments or suggestions?

Mattias comment was to fix the issue and close this report - did you
miss that?

Eli, what is your opinion about my issue #1 (the docstring) btw?


Michael.

Message #61 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Heerdegen <michael_heerdegen <at> web.de>
Cc: mattiase <at> acm.org, monnier <at> iro.umontreal.ca, 69533 <at> debbugs.gnu.org
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Thu, 14 Mar 2024 13:13:27 +0200

> From: Michael Heerdegen <michael_heerdegen <at> web.de>
> Cc: Stefan Monnier <monnier <at> iro.umontreal.ca>,  Mattias Engdegård
>  <mattiase <at> acm.org>,  69533 <at> debbugs.gnu.org
> Date: Thu, 14 Mar 2024 09:44:26 +0100
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > Stefan and Mattias, any comments or suggestions?
> 
> Mattias comment was to fix the issue and close this report - did you
> miss that?

I'd like to hear from Stefan as well.

> Eli, what is your opinion about my issue #1 (the docstring) btw?

I agree.  I also think the way the doc string uses "our" is confusing,
and should be reworded.

Message #64 received at 69533 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Michael Heerdegen <michael_heerdegen <at> web.de>, 69533 <at> debbugs.gnu.org,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Subject: Re: bug#69533: 30.0.50; Wrong byte compilation of a certain apply
 syntax
Date: Thu, 14 Mar 2024 12:18:09 +0100

14 mars 2024 kl. 12.13 skrev Eli Zaretskii <eliz <at> gnu.org>:

>> Eli, what is your opinion about my issue #1 (the docstring) btw?
> 
> I agree.  I also think the way the doc string uses "our" is confusing,
> and should be reworded.

Seems I closed the bug prematurely -- sorry.

Perhaps we should move the example in the `apply` doc string, and that of `funcall`, to shortdoc?

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.