GNU bug report logs - #31549
25.3; bytecompile fails with eval-when-compile

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 31549 in the body.
You can then email your comments to 31549 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ynyaaa <at> gmail.com
To: bug-gnu-emacs <at> gnu.org
Subject: 25.3; bytecompile fails with eval-when-compile
Date: Tue, 22 May 2018 18:00:54 +0900

(byte-compile `(eval-when-compile (list ,@(make-list 2047 0))))

Evaluating the form above, emacs (with -Q option) reports
  Error: Memory exhausted--use C-x s then exit and restart Emacs
in *Compile-Log* buffer.
The returned value of the form is t.




In GNU Emacs 25.3.1 (x86_64-w64-mingw32)
 of 2017-09-27 built on LAPHROAIG
Windowing system distributor 'Microsoft Corp.', version 6.3.9600
Configured using:
 'configure --without-dbus --without-compress-install 'CFLAGS=-O2
 -static -g3' PKG_CONFIG_PATH=/mingw64/lib/pkgconfig'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp932

Major mode: Emacs-Lisp

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns mail-prsvr mail-utils
warnings byte-opt bytecomp byte-compile cl-extra help-mode easymenu
cl-loaddefs pcase cl-lib cconv time-date mule-util japan-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 96577 5188)
 (symbols 56 20186 0)
 (miscs 48 54 117)
 (strings 32 17426 3747)
 (string-bytes 1 488163)
 (vectors 16 13751)
 (vector-slots 8 513163 14478)
 (floats 8 165 49)
 (intervals 56 237 13)
 (buffers 976 19))

Message #8 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: ynyaaa <at> gmail.com
Cc: 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Tue, 22 May 2018 19:33:03 -0400

ynyaaa <at> gmail.com writes:

> (byte-compile `(eval-when-compile (list ,@(make-list 2047 0))))
>
> Evaluating the form above, emacs (with -Q option) reports
>   Error: Memory exhausted--use C-x s then exit and restart Emacs
> in *Compile-Log* buffer.
> The returned value of the form is t.

The error is coming from exec_byte_code:

  if (MAX_ALLOCA / word_size <= XFASTINT (maxdepth))
    memory_full (SIZE_MAX);

It's more like a Lisp stack overflow than a memory exhausted situation
though, hardly calls for restarting Emacs.  Perhaps the byte compiler
should refuse to compile such a large expression?

I can't reproduce in Emacs 26, but only because MAX_ALLOCA is bigger, I
think.

Message #11 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Wed, 23 May 2018 18:08:57 +0300

> Resent-Sender: help-debbugs <at> gnu.org
> From: Noam Postavsky <npostavs <at> gmail.com>
> Date: Tue, 22 May 2018 19:33:03 -0400
> Cc: 31549 <at> debbugs.gnu.org
> 
> > (byte-compile `(eval-when-compile (list ,@(make-list 2047 0))))
> >
> > Evaluating the form above, emacs (with -Q option) reports
> >   Error: Memory exhausted--use C-x s then exit and restart Emacs
> > in *Compile-Log* buffer.
> > The returned value of the form is t.
> 
> The error is coming from exec_byte_code:
> 
>   if (MAX_ALLOCA / word_size <= XFASTINT (maxdepth))
>     memory_full (SIZE_MAX);

You mean, in expansion of SAFE_ALLOCA_LISP_EXTRA?

> It's more like a Lisp stack overflow than a memory exhausted situation
> though, hardly calls for restarting Emacs.  Perhaps the byte compiler
> should refuse to compile such a large expression?

How about simply signaling a special error instead of memory_full?
Something like

  error ("Lisp stack overflow");

Message #14 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Wed, 23 May 2018 18:34:06 -0400

fixed 31549 26.1
quit

Eli Zaretskii <eliz <at> gnu.org> writes:

>> The error is coming from exec_byte_code:
>> 
>>   if (MAX_ALLOCA / word_size <= XFASTINT (maxdepth))
>>     memory_full (SIZE_MAX);
>
> You mean, in expansion of SAFE_ALLOCA_LISP_EXTRA?

Oh, sorry, I missed that the code has been changed quite a bit in Emacs
26.  I said:

    I can't reproduce in Emacs 26, but only because MAX_ALLOCA is
    bigger, I think.

But v26 exec_byte_code will allocate with malloc if needed, so the bug
is fixed (and MAX_ALLOCA is the same size).

> How about simply signaling a special error instead of memory_full?
> Something like
>
>   error ("Lisp stack overflow");

Still might be worth considering changing the error, although stack
overflow doesn't describe it correctly anymore.  As far as I can tell,
SAFE_ALLOCA_LISP_EXTRA will only signal memory_full if the number of
bytes to allocate won't fit in a ptrdiff_t variable.  That should be
pretty much impossible, barring some strange bugs.  So maybe:

--- i/src/lisp.h
+++ w/src/lisp.h
@@ -4662,7 +4662,7 @@ egetenv (const char *var)
     if (INT_MULTIPLY_WRAPV (nelt, word_size, &alloca_nbytes)   \
 	|| INT_ADD_WRAPV (alloca_nbytes, extra, &alloca_nbytes) \
 	|| SIZE_MAX < alloca_nbytes)			       \
-      memory_full (SIZE_MAX);				       \
+      error ("Oversize allocation (0x%lX)", (size_t) alloca_nbytes);    \
     else if (alloca_nbytes <= sa_avail)			       \
       (buf) = AVAIL_ALLOCA (alloca_nbytes);		       \
     else						       \

Message #19 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Thu, 24 May 2018 19:40:12 +0300

> From: Noam Postavsky <npostavs <at> gmail.com>
> Cc: ynyaaa <at> gmail.com,  31549 <at> debbugs.gnu.org
> Date: Wed, 23 May 2018 18:34:06 -0400
> 
> > How about simply signaling a special error instead of memory_full?
> > Something like
> >
> >   error ("Lisp stack overflow");
> 
> Still might be worth considering changing the error, although stack
> overflow doesn't describe it correctly anymore.  As far as I can tell,
> SAFE_ALLOCA_LISP_EXTRA will only signal memory_full if the number of
> bytes to allocate won't fit in a ptrdiff_t variable.  That should be
> pretty much impossible, barring some strange bugs.  So maybe:
> 
> --- i/src/lisp.h
> +++ w/src/lisp.h
> @@ -4662,7 +4662,7 @@ egetenv (const char *var)
>      if (INT_MULTIPLY_WRAPV (nelt, word_size, &alloca_nbytes)   \
>  	|| INT_ADD_WRAPV (alloca_nbytes, extra, &alloca_nbytes) \
>  	|| SIZE_MAX < alloca_nbytes)			       \
> -      memory_full (SIZE_MAX);				       \
> +      error ("Oversize allocation (0x%lX)", (size_t) alloca_nbytes);    \
>      else if (alloca_nbytes <= sa_avail)			       \
>        (buf) = AVAIL_ALLOCA (alloca_nbytes);		       \
>      else						       \

I agree that memory_full is suboptimal here, but "Oversize allocation"
with a number is too technical to be useful to the programmer who
bumps into this problem.  We need some text which will indicate that
the program is too "complex" (a better word is needed here) and should
be simplified.  Can you come up with something along those lines?

Message #22 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Thu, 24 May 2018 17:18:17 -0400

Eli Zaretskii <eliz <at> gnu.org> writes:

>> --- i/src/lisp.h
>> +++ w/src/lisp.h
>> @@ -4662,7 +4662,7 @@ egetenv (const char *var)
>>      if (INT_MULTIPLY_WRAPV (nelt, word_size, &alloca_nbytes)   \
>>  	|| INT_ADD_WRAPV (alloca_nbytes, extra, &alloca_nbytes) \
>>  	|| SIZE_MAX < alloca_nbytes)			       \
>> -      memory_full (SIZE_MAX);				       \
>> +      error ("Oversize allocation (0x%lX)", (size_t) alloca_nbytes);    \
>>      else if (alloca_nbytes <= sa_avail)			       \
>>        (buf) = AVAIL_ALLOCA (alloca_nbytes);		       \
>>      else						       \
>
> I agree that memory_full is suboptimal here, but "Oversize allocation"
> with a number is too technical to be useful to the programmer who
> bumps into this problem.  We need some text which will indicate that
> the program is too "complex" (a better word is needed here) and should
> be simplified.  Can you come up with something along those lines?

Sorry, if my initial response confused things, but I'm fairly certain
now that there is no way to trigger this error by compiling a Lisp
program in Emacs 26.  It would have to require a stack depth of 2^63 (or
2^31 on 32 bit builds), I imagine actual memory exhaustion would happen
first.

Actually, even though memory_full probably isn't correct, maybe we
should just leave it.  Triggering this error probably indicates some bug
in Emacs, so the first thing to do after hitting it would be to set a
breakpoint in gdb; this is a bit more convenient to do with memory_full
than Fsignal or error: fewer false positives.

Message #25 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Fri, 25 May 2018 09:19:19 +0300

> From: Noam Postavsky <npostavs <at> gmail.com>
> Cc: ynyaaa <at> gmail.com,  31549 <at> debbugs.gnu.org
> Date: Thu, 24 May 2018 17:18:17 -0400
> 
> Sorry, if my initial response confused things

Seeking the truth doesn't always work in linear ways ;-)

> I'm fairly certain now that there is no way to trigger this error by
> compiling a Lisp program in Emacs 26.  It would have to require a
> stack depth of 2^63 (or 2^31 on 32 bit builds), I imagine actual
> memory exhaustion would happen first.
> 
> Actually, even though memory_full probably isn't correct, maybe we
> should just leave it.  Triggering this error probably indicates some bug
> in Emacs, so the first thing to do after hitting it would be to set a
> breakpoint in gdb; this is a bit more convenient to do with memory_full
> than Fsignal or error: fewer false positives.

Fine by me, but do we understand what change(s) between 25.3 and 26.1
fixed this problem?  If not, maybe we should try to understand that?

Message #28 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Noam Postavsky <npostavs <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Sun, 27 May 2018 11:09:48 -0400

close 31549
quit

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Noam Postavsky <npostavs <at> gmail.com>
>> Cc: ynyaaa <at> gmail.com,  31549 <at> debbugs.gnu.org
>> Date: Thu, 24 May 2018 17:18:17 -0400
>> 
>> Sorry, if my initial response confused things
>
> Seeking the truth doesn't always work in linear ways ;-)
>
>> I'm fairly certain now that there is no way to trigger this error by
>> compiling a Lisp program in Emacs 26.  It would have to require a
>> stack depth of 2^63 (or 2^31 on 32 bit builds), I imagine actual
>> memory exhaustion would happen first.
>> 
>> Actually, even though memory_full probably isn't correct, maybe we
>> should just leave it.  Triggering this error probably indicates some bug
>> in Emacs, so the first thing to do after hitting it would be to set a
>> breakpoint in gdb; this is a bit more convenient to do with memory_full
>> than Fsignal or error: fewer false positives.
>
> Fine by me, but do we understand what change(s) between 25.3 and 26.1
> fixed this problem?  If not, maybe we should try to understand that?

Ah, here it is, seems pretty straightforward.

[1: cb71a119f7]: 2016-08-09 01:31:22 -0700
  Remove arbitrary limit on bytecode maxdepth
  https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=cb71a119f7231984e010cc28ef33854721036a0f

Message #33 received at 31549 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Noam Postavsky <npostavs <at> gmail.com>
Cc: ynyaaa <at> gmail.com, 31549 <at> debbugs.gnu.org
Subject: Re: bug#31549: 25.3; bytecompile fails with eval-when-compile
Date: Sun, 27 May 2018 19:22:58 +0300

> From: Noam Postavsky <npostavs <at> gmail.com>
> Cc: ynyaaa <at> gmail.com,  31549 <at> debbugs.gnu.org
> Date: Sun, 27 May 2018 11:09:48 -0400
> 
> > Fine by me, but do we understand what change(s) between 25.3 and 26.1
> > fixed this problem?  If not, maybe we should try to understand that?
> 
> Ah, here it is, seems pretty straightforward.
> 
> [1: cb71a119f7]: 2016-08-09 01:31:22 -0700
>   Remove arbitrary limit on bytecode maxdepth

Right, thanks.  Case closed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.