GNU bug report logs - #76546
30.1; Native compilation is corrupting lambdas/closures stored in variables after some time

Previous Next

To reply to this bug, email your comments to 76546 AT debbugs.gnu.org.
There is no need to reopen the bug first.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christian Johansson <christian <at> cvj.se>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.1; Native compilation is corrupting lambdas/closures stored in
 variables after some time
Date: Tue, 25 Feb 2025 11:27:58 +0100

Hello!

There is something fishy going on, but it is a very slippery bug, I have 
trouble reproducing it but it started to happen when native compilation 
was introduced and it has significantly gotten worse with Emacs 30, I 
never happened before native-compilation was introduced.

Maybe you can give me some tips in building a reproducible test case? I 
will try to explain what happens. So prior to Emacs 30 there was only 
one bug I stumbled upon:

1. When a lambda is stored in a variable in a lexical scoped file it 
stores a closure. After some time when native compilation has done its 
magic on the variable you will get an error when you try to output it 
like this:
(format "%S" my-native-compiled-variable) and Emacs crashes.

This happened every time my parser-generator library ran for a long time 
and then tried to output a parser file. The only solution to this was to 
not byte-compile the file that contained the lambdas that was later 
supposed to be printed to a file, this solution worked but it was a 
workaround due to this bug.


2. After Emacs 30 it seems to corrupt strings with escape sequences 
inside lambdas inside variables, like this:

(setq my-lambda (lambda() (looking-at "<\\?=")))

After native-compilation has performed its magic on the variable the 
function body will become corrupted and it will become a invalid 
function body. This happens now to my major my phps-mode, I have 
reproduced the error on two machines with Emacs 30 and it has never 
occurred before Emacs 30 and there has been no recent changes to the 
plugin. If I just eval the function files anew the problem disappears so 
it only happens in compiled files. It however does not happen in 
byte-compiled files, only in native compiled files.


Kind regards
Christian

Message #8 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Christian Johansson <christian <at> cvj.se>, Andrea Corallo <acorallo <at> gnu.org>
Cc: 76546 <at> debbugs.gnu.org
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Wed, 26 Feb 2025 14:43:30 +0200

> Date: Tue, 25 Feb 2025 11:27:58 +0100
> From: Christian Johansson <christian <at> cvj.se>
> 
> There is something fishy going on, but it is a very slippery bug, I have 
> trouble reproducing it but it started to happen when native compilation 
> was introduced and it has significantly gotten worse with Emacs 30, I 
> never happened before native-compilation was introduced.
> 
> Maybe you can give me some tips in building a reproducible test case? I 
> will try to explain what happens. So prior to Emacs 30 there was only 
> one bug I stumbled upon:
> 
> 1. When a lambda is stored in a variable in a lexical scoped file it 
> stores a closure. After some time when native compilation has done its 
> magic on the variable you will get an error when you try to output it 
> like this:
> (format "%S" my-native-compiled-variable) and Emacs crashes.
> 
> This happened every time my parser-generator library ran for a long time 
> and then tried to output a parser file. The only solution to this was to 
> not byte-compile the file that contained the lambdas that was later 
> supposed to be printed to a file, this solution worked but it was a 
> workaround due to this bug.
> 
> 
> 2. After Emacs 30 it seems to corrupt strings with escape sequences 
> inside lambdas inside variables, like this:
> 
> (setq my-lambda (lambda() (looking-at "<\\?=")))
> 
> After native-compilation has performed its magic on the variable the 
> function body will become corrupted and it will become a invalid 
> function body. This happens now to my major my phps-mode, I have 
> reproduced the error on two machines with Emacs 30 and it has never 
> occurred before Emacs 30 and there has been no recent changes to the 
> plugin. If I just eval the function files anew the problem disappears so 
> it only happens in compiled files. It however does not happen in 
> byte-compiled files, only in native compiled files.

Andrea, any comments or suggestions?

Message #11 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Christian Johansson <christian <at> cvj.se>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 76546 <at> debbugs.gnu.org, Andrea Corallo <acorallo <at> gnu.org>
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Wed, 26 Feb 2025 21:55:28 +0100

Today the error disappeared on both machines without any changes in code, it makes no sense really but if I could trigger native compilation I could do some testing back and forth. It can be a issue with the garbage collection mechanism also, the things could be dependent on each other as well. Maybe only native compiled + byte compiled + garbage collected variables get corrupted sometimes?

Non reproducible bugs that only happen after a significant period of time.. variable data gets corrupted, that is at least true

Kind Regards
Christian

> On 26 Feb 2025, at 14:03, Eli Zaretskii <eliz <at> gnu.org> wrote:
> 
> 
>> 
>> Date: Tue, 25 Feb 2025 11:27:58 +0100
>> From: Christian Johansson <christian <at> cvj.se>
>> 
>> There is something fishy going on, but it is a very slippery bug, I have
>> trouble reproducing it but it started to happen when native compilation
>> was introduced and it has significantly gotten worse with Emacs 30, I
>> never happened before native-compilation was introduced.
>> 
>> Maybe you can give me some tips in building a reproducible test case? I
>> will try to explain what happens. So prior to Emacs 30 there was only
>> one bug I stumbled upon:
>> 
>> 1. When a lambda is stored in a variable in a lexical scoped file it
>> stores a closure. After some time when native compilation has done its
>> magic on the variable you will get an error when you try to output it
>> like this:
>> (format "%S" my-native-compiled-variable) and Emacs crashes.
>> 
>> This happened every time my parser-generator library ran for a long time
>> and then tried to output a parser file. The only solution to this was to
>> not byte-compile the file that contained the lambdas that was later
>> supposed to be printed to a file, this solution worked but it was a
>> workaround due to this bug.
>> 
>> 
>> 2. After Emacs 30 it seems to corrupt strings with escape sequences
>> inside lambdas inside variables, like this:
>> 
>> (setq my-lambda (lambda() (looking-at "<\\?=")))
>> 
>> After native-compilation has performed its magic on the variable the
>> function body will become corrupted and it will become a invalid
>> function body. This happens now to my major my phps-mode, I have
>> reproduced the error on two machines with Emacs 30 and it has never
>> occurred before Emacs 30 and there has been no recent changes to the
>> plugin. If I just eval the function files anew the problem disappears so
>> it only happens in compiled files. It however does not happen in
>> byte-compiled files, only in native compiled files.
> 
> Andrea, any comments or suggestions?

Message #14 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Christian Johansson <christian <at> cvj.se>
Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Thu, 27 Feb 2025 10:02:41 +0200

> From: Christian Johansson <christian <at> cvj.se>
> Date: Wed, 26 Feb 2025 21:55:28 +0100
> Cc: Andrea Corallo <acorallo <at> gnu.org>, 76546 <at> debbugs.gnu.org
> 
> Today the error disappeared on both machines without any changes in code, it makes no sense really but if I could trigger native compilation I could do some testing back and forth. It can be a issue with the garbage collection mechanism also, the things could be dependent on each other as well. Maybe only native compiled + byte compiled + garbage collected variables get corrupted sometimes?

If you need a way to trigger native compilation, then removing the
corresponding .eln file(s) should do it.

Message #17 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <acorallo <at> gnu.org>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Christian Johansson <christian <at> cvj.se>, 76546 <at> debbugs.gnu.org
Subject: Re: bug#76546: 30.1; Native compilation is corrupting
 lambdas/closures stored in variables after some time
Date: Mon, 14 Apr 2025 05:59:13 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Date: Tue, 25 Feb 2025 11:27:58 +0100
>> From: Christian Johansson <christian <at> cvj.se>
>> 
>> There is something fishy going on, but it is a very slippery bug, I have 
>> trouble reproducing it but it started to happen when native compilation 
>> was introduced and it has significantly gotten worse with Emacs 30, I 
>> never happened before native-compilation was introduced.
>> 
>> Maybe you can give me some tips in building a reproducible test case? I 
>> will try to explain what happens. So prior to Emacs 30 there was only 
>> one bug I stumbled upon:
>> 
>> 1. When a lambda is stored in a variable in a lexical scoped file it 
>> stores a closure. After some time when native compilation has done its 
>> magic on the variable you will get an error when you try to output it 
>> like this:
>> (format "%S" my-native-compiled-variable) and Emacs crashes.
>> 
>> This happened every time my parser-generator library ran for a long time 
>> and then tried to output a parser file. The only solution to this was to 
>> not byte-compile the file that contained the lambdas that was later 
>> supposed to be printed to a file, this solution worked but it was a 
>> workaround due to this bug.
>> 
>> 
>> 2. After Emacs 30 it seems to corrupt strings with escape sequences 
>> inside lambdas inside variables, like this:
>> 
>> (setq my-lambda (lambda() (looking-at "<\\?=")))
>> 
>> After native-compilation has performed its magic on the variable the 
>> function body will become corrupted and it will become a invalid 
>> function body. This happens now to my major my phps-mode, I have 
>> reproduced the error on two machines with Emacs 30 and it has never 
>> occurred before Emacs 30 and there has been no recent changes to the 
>> plugin. If I just eval the function files anew the problem disappears so 
>> it only happens in compiled files. It however does not happen in 
>> byte-compiled files, only in native compiled files.
>
> Andrea, any comments or suggestions?

I can't reproduce this on master.  I've:

test.el:
======
;; -*- lexical-binding: t -*-                                                        
(setq my-lambda (lambda() (looking-at "<\\?=")))
======

If I run emacs:
$ emacs --batch -eval '(progn (load (native-compile "/home/andcor03/test/test.el")) (funcall my-lambda))'

Seems to work for me.  Not sure if the reproducer should be something
different tho.

  Andrea

Message #20 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Christian Johansson <christian <at> cvj.se>
To: Andrea Corallo <acorallo <at> gnu.org>
Cc: 76546 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Tue, 15 Apr 2025 09:58:07 +0200

Hi

The initial issue disappeared by itself without any updates to software, it started to happen after Emacs upgrade on two different machines but resolved itself. Must be something related to corrupt storage of byte compiled or native compiled code

I have not encountered the issue since but other strange issues like keymap rebinding after some time in a Emacs session

I have one issue that is reproducible but only after a couple of days of Emacs running and exahausting the memory, in that case the lambda stored in variable becomes binary and printing it crashes Emacs

Regards
Christian

> Andrea Corallo <acorallo <at> gnu.org> kirjoitti 2025-04-14 kello 12.15:
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
>>> Date: Tue, 25 Feb 2025 11:27:58 +0100
>>> From: Christian Johansson <christian <at> cvj.se>
>>> 
>>> There is something fishy going on, but it is a very slippery bug, I have
>>> trouble reproducing it but it started to happen when native compilation
>>> was introduced and it has significantly gotten worse with Emacs 30, I
>>> never happened before native-compilation was introduced.
>>> 
>>> Maybe you can give me some tips in building a reproducible test case? I
>>> will try to explain what happens. So prior to Emacs 30 there was only
>>> one bug I stumbled upon:
>>> 
>>> 1. When a lambda is stored in a variable in a lexical scoped file it
>>> stores a closure. After some time when native compilation has done its
>>> magic on the variable you will get an error when you try to output it
>>> like this:
>>> (format "%S" my-native-compiled-variable) and Emacs crashes.
>>> 
>>> This happened every time my parser-generator library ran for a long time
>>> and then tried to output a parser file. The only solution to this was to
>>> not byte-compile the file that contained the lambdas that was later
>>> supposed to be printed to a file, this solution worked but it was a
>>> workaround due to this bug.
>>> 
>>> 
>>> 2. After Emacs 30 it seems to corrupt strings with escape sequences
>>> inside lambdas inside variables, like this:
>>> 
>>> (setq my-lambda (lambda() (looking-at "<\\?=")))
>>> 
>>> After native-compilation has performed its magic on the variable the
>>> function body will become corrupted and it will become a invalid
>>> function body. This happens now to my major my phps-mode, I have
>>> reproduced the error on two machines with Emacs 30 and it has never
>>> occurred before Emacs 30 and there has been no recent changes to the
>>> plugin. If I just eval the function files anew the problem disappears so
>>> it only happens in compiled files. It however does not happen in
>>> byte-compiled files, only in native compiled files.
>> 
>> Andrea, any comments or suggestions?
> 
> I can't reproduce this on master.  I've:
> 
> test.el:
> ======
> ;; -*- lexical-binding: t -*-                                                        
> (setq my-lambda (lambda() (looking-at "<\\?=")))
> ======
> 
> If I run emacs:
> $ emacs --batch -eval '(progn (load (native-compile "/home/andcor03/test/test.el")) (funcall my-lambda))'
> 
> Seems to work for me.  Not sure if the reproducer should be something
> different tho.
> 
>  Andrea

Message #23 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Christian Johansson <christian <at> cvj.se>
Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Tue, 15 Apr 2025 11:38:34 +0300

> From: Christian Johansson <christian <at> cvj.se>
> Date: Tue, 15 Apr 2025 09:58:07 +0200
> Cc: Eli Zaretskii <eliz <at> gnu.org>, 76546 <at> debbugs.gnu.org
> 
> I have one issue that is reproducible but only after a couple of days of Emacs running and exahausting the memory, in that case the lambda stored in variable becomes binary and printing it crashes Emacs

If the value cannot be printed, how do you know that it is binary?

Could you perhaps run Emacs under GDB, so that when this happens, you
could show the value and perhaps invoke some other commands we would
like you to try and show the results?

Message #26 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: christian <at> cvj.se
Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Sat, 26 Apr 2025 14:45:41 +0300

> Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
> Date: Tue, 15 Apr 2025 11:38:34 +0300
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Christian Johansson <christian <at> cvj.se>
> > Date: Tue, 15 Apr 2025 09:58:07 +0200
> > Cc: Eli Zaretskii <eliz <at> gnu.org>, 76546 <at> debbugs.gnu.org
> > 
> > I have one issue that is reproducible but only after a couple of days of Emacs running and exahausting the memory, in that case the lambda stored in variable becomes binary and printing it crashes Emacs
> 
> If the value cannot be printed, how do you know that it is binary?
> 
> Could you perhaps run Emacs under GDB, so that when this happens, you
> could show the value and perhaps invoke some other commands we would
> like you to try and show the results?

Ping!  Any further information or progress with this?

Message #29 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Christian Johansson <christian <at> cvj.se>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Sat, 26 Apr 2025 15:09:07 +0200

Yes maybe, it was apparent from the error message that the function had unexpected data and trying to print it crashed Emacs.. I will check for an old log

But this error has been present for many years so it is not a new thing, I solved it by not byte compilling some files

> Eli Zaretskii <eliz <at> gnu.org> kirjoitti 2025-04-26 kello 13.45:
> 
> 
>> 
>> Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
>> Date: Tue, 15 Apr 2025 11:38:34 +0300
>> From: Eli Zaretskii <eliz <at> gnu.org>
>> 
>>> From: Christian Johansson <christian <at> cvj.se>
>>> Date: Tue, 15 Apr 2025 09:58:07 +0200
>>> Cc: Eli Zaretskii <eliz <at> gnu.org>, 76546 <at> debbugs.gnu.org
>>> 
>>> I have one issue that is reproducible but only after a couple of days of Emacs running and exahausting the memory, in that case the lambda stored in variable becomes binary and printing it crashes Emacs
>> 
>> If the value cannot be printed, how do you know that it is binary?
>> 
>> Could you perhaps run Emacs under GDB, so that when this happens, you
>> could show the value and perhaps invoke some other commands we would
>> like you to try and show the results?
> 
> Ping!  Any further information or progress with this?

Message #32 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Christian Johansson <christian <at> cvj.se>, acorallo <at> gnu.org
Cc: 76546 <at> debbugs.gnu.org
Subject: Re: bug#76546: 30.1;
 Native compilation is corrupting lambdas/closures stored in variables
 after some time
Date: Sat, 10 May 2025 12:31:24 +0300

Ping!  Any further information or ideas that could allow us to make
progress here?

> From: Christian Johansson <christian <at> cvj.se>
> Date: Sat, 26 Apr 2025 15:09:07 +0200
> Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
> 
> Yes maybe, it was apparent from the error message that the function had unexpected data and trying to print it crashed Emacs.. I will check for an old log
> 
> But this error has been present for many years so it is not a new thing, I solved it by not byte compilling some files
> 
> > Eli Zaretskii <eliz <at> gnu.org> kirjoitti 2025-04-26 kello 13.45:
> > 
> > 
> >> 
> >> Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
> >> Date: Tue, 15 Apr 2025 11:38:34 +0300
> >> From: Eli Zaretskii <eliz <at> gnu.org>
> >> 
> >>> From: Christian Johansson <christian <at> cvj.se>
> >>> Date: Tue, 15 Apr 2025 09:58:07 +0200
> >>> Cc: Eli Zaretskii <eliz <at> gnu.org>, 76546 <at> debbugs.gnu.org
> >>> 
> >>> I have one issue that is reproducible but only after a couple of days of Emacs running and exahausting the memory, in that case the lambda stored in variable becomes binary and printing it crashes Emacs
> >> 
> >> If the value cannot be printed, how do you know that it is binary?
> >> 
> >> Could you perhaps run Emacs under GDB, so that when this happens, you
> >> could show the value and perhaps invoke some other commands we would
> >> like you to try and show the results?
> > 
> > Ping!  Any further information or progress with this?
> 

Message #35 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Christian Johansson <christian <at> cvj.se>
To: Eli Zaretskii <eliz <at> gnu.org>, acorallo <at> gnu.org
Cc: 76546 <at> debbugs.gnu.org
Subject: Re: bug#76546: 30.1; Native compilation is corrupting
 lambdas/closures stored in variables after some time
Date: Thu, 15 May 2025 06:07:15 +0200

I have a discussion with Stefan Monnier about this, this seems to be a 
feature and not a bug, the printing of lambdas has been limited in new 
Emacs version to avoid this issue I think

On 5/10/25 11:31, Eli Zaretskii wrote:
> Ping!  Any further information or ideas that could allow us to make
> progress here?
>
>> From: Christian Johansson <christian <at> cvj.se>
>> Date: Sat, 26 Apr 2025 15:09:07 +0200
>> Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
>>
>> Yes maybe, it was apparent from the error message that the function had unexpected data and trying to print it crashed Emacs.. I will check for an old log
>>
>> But this error has been present for many years so it is not a new thing, I solved it by not byte compilling some files
>>
>>> Eli Zaretskii <eliz <at> gnu.org> kirjoitti 2025-04-26 kello 13.45:
>>>
>>> 
>>>> Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
>>>> Date: Tue, 15 Apr 2025 11:38:34 +0300
>>>> From: Eli Zaretskii <eliz <at> gnu.org>
>>>>
>>>>> From: Christian Johansson <christian <at> cvj.se>
>>>>> Date: Tue, 15 Apr 2025 09:58:07 +0200
>>>>> Cc: Eli Zaretskii <eliz <at> gnu.org>, 76546 <at> debbugs.gnu.org
>>>>>
>>>>> I have one issue that is reproducible but only after a couple of days of Emacs running and exahausting the memory, in that case the lambda stored in variable becomes binary and printing it crashes Emacs
>>>> If the value cannot be printed, how do you know that it is binary?
>>>>
>>>> Could you perhaps run Emacs under GDB, so that when this happens, you
>>>> could show the value and perhaps invoke some other commands we would
>>>> like you to try and show the results?
>>> Ping!  Any further information or progress with this?

Message #38 received at 76546 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Christian Johansson <christian <at> cvj.se>,
 Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 76546 <at> debbugs.gnu.org, acorallo <at> gnu.org
Subject: Re: bug#76546: 30.1; Native compilation is corrupting
 lambdas/closures stored in variables after some time
Date: Sat, 17 May 2025 12:42:14 +0300

> Date: Thu, 15 May 2025 06:07:15 +0200
> Cc: 76546 <at> debbugs.gnu.org
> From: Christian Johansson <christian <at> cvj.se>
> 
> I have a discussion with Stefan Monnier about this, this seems to be a 
> feature and not a bug, the printing of lambdas has been limited in new 
> Emacs version to avoid this issue I think

Was the discussion with Stefan off-list?

Stefan, should I close this bug, then?

Message #43 received at 76546-done <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Christian Johansson <christian <at> cvj.se>, acorallo <at> gnu.org,
 76546-done <at> debbugs.gnu.org
Subject: Re: bug#76546: 30.1; Native compilation is corrupting
 lambdas/closures stored in variables after some time
Date: Tue, 20 May 2025 13:26:48 -0400

>> I have a discussion with Stefan Monnier about this, this seems to be a
>> feature and not a bug, the printing of lambdas has been limited in new
>> Emacs version to avoid this issue I think
> Was the discussion with Stefan off-list?

My best guess is that he's referring to

    https://lists.gnu.org/archive/html/emacs-devel/2025-05/msg00133.html

> Stefan, should I close this bug, then?

I think so, mostly because there is no actionable element in it: no
clear description of what was the problem and even less info about how
to reproduce it.


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.