GNU bug report logs - #76868
29.4; cperl-mode __END__ is incorrectly handled

Previous Next

To reply to this bug, email your comments to 76868 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Ciolfi <ciolfi <at> mathworks.com>
To: <bug-gnu-emacs <at> gnu.org>
Subject: 29.4; cperl-mode __END__ is incorrectly handled
Date: Sat, 8 Mar 2025 10:34:20 -0500

Hi

Given the following cperl-mode program, test.pl

    # -*- mode: cperl; -*-
    print "hello\n";
    __END__
    This is not
      perl code!

The lines after the __END__ are treated as perl code. They are syntactically colored and indented
which is incorrect. In perl, __END__ defines the end of the Perl code in the file. Any text that
appears after __END__ is ignored by the Perl compiler.

All lines after __END__ should be treated as comments.

Thanks
John




In GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38,
 cairo version 1.16.0) of 2024-07-22, modified by Debian built on
 x86-ubc-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12101006
System Description: Debian GNU/Linux 12 (bookworm)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --with-cairo --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/reproducible-path/emacs-29.4+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2
XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils comp comp-cstr
warnings subr-x rx cl-seq cl-macs gv cl-extra help-mode bytecomp
byte-compile cus-edit pp cus-start cus-load icons wid-edit cl-loaddefs
cl-lib rmc iso-transl tooltip cconv eldoc paren electric uniquify
ediff-hook vc-hooks lisp-float-type elisp-mode mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting cairo move-toolbar gtk x-toolkit xinput2 x multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 105675 8530)
 (symbols 48 8853 7)
 (strings 32 23487 2243)
 (string-bytes 1 691014)
 (vectors 16 17555)
 (vector-slots 8 353713 18074)
 (floats 8 39 42)
 (intervals 56 579 2)
 (buffers 984 14))

Message #8 received at 76868 <at> debbugs.gnu.org (full text, mbox):

From: Mauro Aranda <maurooaranda <at> gmail.com>
To: ciolfi <at> mathworks.com, 76868 <at> debbugs.gnu.org
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
Date: Sat, 8 Mar 2025 12:51:16 -0300

On 8/3/25 12:34, John Ciolfi via Bug reports for GNU Emacs, the Swiss 
army knife of text editors wrote:
>
> Hi
>
> Given the following cperl-mode program, test.pl
>
>      # -*- mode: cperl; -*-
>      print "hello\n";
>      __END__
>      This is not
>        perl code!
>
> The lines after the __END__ are treated as perl code. They are 
syntactically colored and indented
> which is incorrect. In perl, __END__ defines the end of the Perl code 
in the file. Any text that
> appears after __END__ is ignored by the Perl compiler.
>
> All lines after __END__ should be treated as comments.
>

Any chance that the option cperl-fontify-trailer helps here?

I remember reporting something similar in:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66161

Message #11 received at 76868 <at> debbugs.gnu.org (full text, mbox):

From: John Ciolfi <ciolfi <at> mathworks.com>
To: Mauro Aranda <maurooaranda <at> gmail.com>, "76868 <at> debbugs.gnu.org"
 <76868 <at> debbugs.gnu.org>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
Date: Sun, 9 Mar 2025 02:59:11 +0000

[Message part 1 (text/plain, inline)]

Yes, setting cperl-fontify-trailer to comment fixes the bug. In my case, the lines after the __END__ are more than just comments and get corrupted when they are indented. There is also the __DATA__ keyword and the default setting of cperl-fontify-trailier causes the data to be indented which alters the behavior of the program (corrupts the result). Consider:

while (my $line = <DATA>) { print $line; }
close DATA;
__DATA__
This is not
perl code

If you indent, C-x h followed by C-M-\, the results change. Indenting shouldn't change results.

Could you consider changing the default value of cperl-fontify-trailer to comment so that cperl-mode doesn't indent them? Also, the name is a little misleading because the more important piece is whether to semantically treat the lines after these keywords as perl code. The Perl interpreter never treats these as perl code, so it's odd cperl-mode treats them as perl code. The only case I'm aware of where the lines following these keywords is perl is to use the trick to put __END__ to comment out the bottom part of a perl program, but even in that case, it should be colored as comments.

Thanks

________________________________
From: Mauro Aranda <maurooaranda <at> gmail.com>
Sent: Saturday, March 8, 2025 10:51 AM
To: John Ciolfi <ciolfi <at> mathworks.com>; 76868 <at> debbugs.gnu.org <76868 <at> debbugs.gnu.org>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled

On 8/3/25 12:34, John Ciolfi via Bug reports for GNU Emacs, the Swiss
army knife of text editors wrote:
>
> Hi
>
> Given the following cperl-mode program, test.pl
>
> # -*- mode: cperl; -*-
> print "hello\n";
> __END__
> This is not
> perl code!
>
> The lines after the __END__ are treated as perl code. They are
syntactically colored and indented
> which is incorrect. In perl, __END__ defines the end of the Perl code
in the file. Any text that
> appears after __END__ is ignored by the Perl compiler.
>
> All lines after __END__ should be treated as comments.
>

Any chance that the option cperl-fontify-trailer helps here?

I remember reporting something similar in:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66161<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66161>

[Message part 2 (text/html, inline)]

Message #14 received at 76868 <at> debbugs.gnu.org (full text, mbox):

From: Mauro Aranda <maurooaranda <at> gmail.com>
To: John Ciolfi <ciolfi <at> mathworks.com>,
 "76868 <at> debbugs.gnu.org" <76868 <at> debbugs.gnu.org>
Cc: Harald Jörg <haj <at> posteo.de>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
Date: Sun, 9 Mar 2025 08:00:42 -0300

On 8/3/25 23:59, John Ciolfi wrote:
> Yes, setting cperl-fontify-trailer to comment fixes the bug. In my
> case, the lines after the __END__ are more than just comments and get
> corrupted when they are indented. There is also the __DATA__ keyword
> and the default setting of cperl-fontify-trailier causes the data to
> be indented which alters the behavior of the program (corrupts the
> result). Consider:
>
> while (my $line = <DATA>) { print $line; } close DATA; __DATA__ This
> is not perl code
>
> If you indent, C-x h followed by C-M-\, the results change. Indenting
> shouldn't change results.
>
> Could you consider changing the default value of cperl-fontify-trailer
> to comment so that cperl-mode doesn't indent them? Also, the name is a
> little misleading because the more important piece is whether to
> semantically treat the lines after these keywords as perl code. The
> Perl interpreter never treats these as perl code, so it's odd
> cperl-mode treats them as perl code. The only case I'm aware of where
> the lines following these keywords is perl is to use the trick to put
> __END__ to comment out the bottom part of a perl program, but even in
> that case, it should be colored as comments.
>
> Thanks
>

CCing Harald.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Harald Jörg <haj <at> posteo.de>
To: John Ciolfi via "Bug reports for GNU Emacs, the Swiss army knife of text
 editors" <bug-gnu-emacs <at> gnu.org>
Cc: "76868 <at> debbugs.gnu.org" <76868 <at> debbugs.gnu.org>,
 John Ciolfi <ciolfi <at> mathworks.com>, Mauro Aranda <maurooaranda <at> gmail.com>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
Date: Sun, 09 Mar 2025 14:58:13 +0000

John Ciolfi via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs <at> gnu.org> writes:

> Yes, setting cperl-fontify-trailer to comment fixes the bug. In my
> case, the lines after the __END__ are more than just comments and get
> corrupted when they are indented.  There is also the __DATA__ keyword
> and the default setting of cperl-fontify-trailier causes the data to
> be indented which alters the behavior of the program (corrupts the
> result). Consider:
>
> while (my $line = <DATA>) { print $line; }
> close DATA;
> __DATA__
> This is not
> perl code
>
> If you indent, C-x h followed by C-M-\, the results change. Indenting
> shouldn't change results.

Indeed, setting the option cperl-fontify-trailer to 'comment' is
supposed to prevent this sort of thing from happening.

> Could you consider changing the default value of cperl-fontify-trailer
> to comment so that cperl-mode doesn't indent them? Also, the name is a
> little misleading because the more important piece is whether to
> semantically treat the lines after these keywords as perl code. ...

I agree that the name of the option is misleading.  Maybe we should add
an alias like cperl-trailer-type?  The possible values of "perl-code"
and "comment" still work with that name.  However, changing the default
is, in my opinion, not the correct way, because there are valid use
cases for the current default behavior.

> ... The Perl interpreter never treats these as perl code, so it's odd
> cperl-mode treats them as perl code. ...

That's not the whole truth.  Programs which use AutoSplit or SelfLoader
work by intentionally having Perl code after the __END__ or __DATA__
tokens.  Both AutoSplit and SelfLoader are part of the Perl core.

   https://perldoc.perl.org/AutoSplit
   https://perldoc.perl.org/SelfLoader

> The only case I'm aware of where the lines following these keywords is
> perl is to use the trick to put __END__ to comment out the bottom part
> of a perl program, but even in that case, it should be colored as
> comments.

By far more common is the practice to have POD (Perl's documentation
format) after the __END__ token.  This is not read by the Perl
interpreter, but by perldoc and other POD viewers.  POD is part of the
Perl language.  CPerl mode handles POD for fontification, and supports
POD authoring by imenu indexing of POD headings and by ispell checking
for POD sections.

So, different ways to use the space after the tokens are valid.  If you
do not use POD or Perl code after __END__ or __DATA__, setting the
option cperl-fontify-trailer to "comment" does the trick.  But the
default value is there for a valid purpose, too.  User options are not
supposed to change the behavior of Emacs which would be the case if the
default value is flipped.

If you regularly indent a whole buffer which may contain trailers not
understood by CPerl mode, you can set the variable temporarily:

  (defun indent-buffer-without-trailer ()
    (interactive)
    (let ((cperl-fontify-trailer "comment"))
      (cperl-indent-region (point-min) (point-max))))

> Thanks
>
> ------------------------------------------------------------------------------------
> From: Mauro Aranda <maurooaranda <at> gmail.com>
> Sent: Saturday, March 8, 2025 10:51 AM
> To: John Ciolfi <ciolfi <at> mathworks.com>; 76868 <at> debbugs.gnu.org
> <76868 <at> debbugs.gnu.org>
> Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
>  
> On 8/3/25 12:34, John Ciolfi via Bug reports for GNU Emacs, the Swiss
> army knife of text editors wrote:
>>
>> Hi
>>
>> Given the following cperl-mode program, test.pl
>>
>> # -*- mode: cperl; -*-
>> print "hello\n";
>> __END__
>> This is not
>> perl code!
>>
>> The lines after the __END__ are treated as perl code. They are
> syntactically colored and indented
>> which is incorrect. In perl, __END__ defines the end of the Perl code
> in the file. Any text that
>> appears after __END__ is ignored by the Perl compiler.
>>
>> All lines after __END__ should be treated as comments.
>>
>
> Any chance that the option cperl-fontify-trailer helps here?
>
> I remember reporting something similar in:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66161

In the comments of that bug report I've also listed CPAN examples for
each of the use cases.
-- 
Cheers,
haj

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Ciolfi <ciolfi <at> mathworks.com>
To: Harald Jörg <haj <at> posteo.de>, "John Ciolfi via Bug
 reports for GNU Emacs, the Swiss army knife of text editors"
 <bug-gnu-emacs <at> gnu.org>
Cc: "76868 <at> debbugs.gnu.org" <76868 <at> debbugs.gnu.org>,
 Mauro Aranda <maurooaranda <at> gmail.com>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
Date: Sun, 9 Mar 2025 16:41:00 +0000

[Message part 1 (text/plain, inline)]

Thanks for the detailed response. I learned something new. To me it seems that perl isn't supplying enough information for cperl-mode to get the trailers correct. Ideally, we could definitively tell the type of content below the __END__ or __DATA__ keywords.  I suppose one could set file local variables to configure cperl-fontify-trailer. However, one needs to know about this option. I wonder if there's a way to make the option more obvious?

A small suggestion is to update the help text or cperl-fontify-trailer to indicate it also changes the indent behavior.  I think the name cperl-trailer-type is a good name too, though updating the help text may be sufficient.

Thanks
John

________________________________
From: Harald Jörg <haj <at> posteo.de>
Sent: Sunday, March 9, 2025 8:28 PM
To: John Ciolfi via Bug reports for GNU Emacs, the Swiss army knife of text editors <bug-gnu-emacs <at> gnu.org>
Cc: Mauro Aranda <maurooaranda <at> gmail.com>; 76868 <at> debbugs.gnu.org <76868 <at> debbugs.gnu.org>; John Ciolfi <ciolfi <at> mathworks.com>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled

John Ciolfi via "Bug reports for GNU Emacs, the Swiss army knife of text
editors" <bug-gnu-emacs <at> gnu.org> writes:

> Yes, setting cperl-fontify-trailer to comment fixes the bug. In my
> case, the lines after the __END__ are more than just comments and get
> corrupted when they are indented. There is also the __DATA__ keyword
> and the default setting of cperl-fontify-trailier causes the data to
> be indented which alters the behavior of the program (corrupts the
> result). Consider:
>
> while (my $line = <DATA>) { print $line; }
> close DATA;
> __DATA__
> This is not
> perl code
>
> If you indent, C-x h followed by C-M-\, the results change. Indenting
> shouldn't change results.

Indeed, setting the option cperl-fontify-trailer to 'comment' is
supposed to prevent this sort of thing from happening.

> Could you consider changing the default value of cperl-fontify-trailer
> to comment so that cperl-mode doesn't indent them? Also, the name is a
> little misleading because the more important piece is whether to
> semantically treat the lines after these keywords as perl code. ...

I agree that the name of the option is misleading. Maybe we should add
an alias like cperl-trailer-type? The possible values of "perl-code"
and "comment" still work with that name. However, changing the default
is, in my opinion, not the correct way, because there are valid use
cases for the current default behavior.

> ... The Perl interpreter never treats these as perl code, so it's odd
> cperl-mode treats them as perl code. ...

That's not the whole truth. Programs which use AutoSplit or SelfLoader
work by intentionally having Perl code after the __END__ or __DATA__
tokens. Both AutoSplit and SelfLoader are part of the Perl core.

https://perldoc.perl.org/AutoSplit<https://perldoc.perl.org/AutoSplit>
https://perldoc.perl.org/SelfLoader<https://perldoc.perl.org/SelfLoader>

> The only case I'm aware of where the lines following these keywords is
> perl is to use the trick to put __END__ to comment out the bottom part
> of a perl program, but even in that case, it should be colored as
> comments.

By far more common is the practice to have POD (Perl's documentation
format) after the __END__ token. This is not read by the Perl
interpreter, but by perldoc and other POD viewers. POD is part of the
Perl language. CPerl mode handles POD for fontification, and supports
POD authoring by imenu indexing of POD headings and by ispell checking
for POD sections.

So, different ways to use the space after the tokens are valid. If you
do not use POD or Perl code after __END__ or __DATA__, setting the
option cperl-fontify-trailer to "comment" does the trick. But the
default value is there for a valid purpose, too. User options are not
supposed to change the behavior of Emacs which would be the case if the
default value is flipped.

If you regularly indent a whole buffer which may contain trailers not
understood by CPerl mode, you can set the variable temporarily:

(defun indent-buffer-without-trailer ()
(interactive)
(let ((cperl-fontify-trailer "comment"))
(cperl-indent-region (point-min) (point-max))))

> Thanks
>
> ------------------------------------------------------------------------------------
> From: Mauro Aranda <maurooaranda <at> gmail.com>
> Sent: Saturday, March 8, 2025 10:51 AM
> To: John Ciolfi <ciolfi <at> mathworks.com>; 76868 <at> debbugs.gnu.org
> <76868 <at> debbugs.gnu.org>
> Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
>
> On 8/3/25 12:34, John Ciolfi via Bug reports for GNU Emacs, the Swiss
> army knife of text editors wrote:
>>
>> Hi
>>
>> Given the following cperl-mode program, test.pl
>>
>> # -*- mode: cperl; -*-
>> print "hello\n";
>> __END__
>> This is not
>> perl code!
>>
>> The lines after the __END__ are treated as perl code. They are
> syntactically colored and indented
>> which is incorrect. In perl, __END__ defines the end of the Perl code
> in the file. Any text that
>> appears after __END__ is ignored by the Perl compiler.
>>
>> All lines after __END__ should be treated as comments.
>>
>
> Any chance that the option cperl-fontify-trailer helps here?
>
> I remember reporting something similar in:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66161<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=66161>

In the comments of that bug report I've also listed CPAN examples for
each of the use cases.
--
Cheers,
haj

[Message part 2 (text/html, inline)]

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Harald Jörg <haj <at> posteo.de>
To: John Ciolfi <ciolfi <at> mathworks.com>
Cc: "76868 <at> debbugs.gnu.org" <76868 <at> debbugs.gnu.org>,
 "John Ciolfi via Bug reports for GNU Emacs, the Swiss army knife of text
 editors" <bug-gnu-emacs <at> gnu.org>, Mauro Aranda <maurooaranda <at> gmail.com>
Subject: Re: bug#76868: 29.4; cperl-mode __END__ is incorrectly handled
Date: Mon, 10 Mar 2025 00:34:59 +0000

John Ciolfi <ciolfi <at> mathworks.com> writes:

> Thanks for the detailed response. I learned something new. To me it
> seems that perl isn't supplying enough information for cperl-mode to
> get the trailers correct. Ideally, we could definitively tell the type
> of content below the __END__ or __DATA__ keywords.  I suppose one
> could set file local variables to configure
> cperl-fontify-trailer. However, one needs to know about this option. I
> wonder if there's a way to make the option more obvious?

File local variables work but need caution.  Several tools might compete
with content for the first line, and a local variables list "near the
end of the file" might come unexpected to a program reading from the
DATA file handle.  Often it is "less intrusive" to use a .dir-locals.el
file.

> A small suggestion is to update the help text or cperl-fontify-trailer
> to indicate it also changes the indent behavior.  I think the name
> cperl-trailer-type is a good name too, though updating the help text
> may be sufficient.

Improving the docstring by including the behavior of indentation is
indeed a good idea!  I'll do that for the master branch.
-- 
Cheers,
haj

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.