GNU bug report logs - #52496
syntax highlighting problem on emacs shell-script mode

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 52496 in the body.
You can then email your comments to 52496 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: syntax highlighting problem on emacs shell-script mode
Date: Wed, 15 Dec 2021 03:35:24 +0300

I don't know exactly how to explain it but emacs comments most of a
script if the script contains too many long strings. As an example
look at the syntax highlighting of the script at
https://gitlab.com/uppercat/tlbx/-/raw/master/tlbx and compare the
syntax highlighting for it in vim and emacs.

Message #8 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Cc: 52496 <at> debbugs.gnu.org
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 13:54:50 +0100

[Message part 1 (text/plain, inline)]

Yilkal Argaw <yilkalargawworkneh <at> gmail.com> writes:

> I don't know exactly how to explain it but emacs comments most of a
> script if the script contains too many long strings. As an example
> look at the syntax highlighting of the script at
> https://gitlab.com/uppercat/tlbx/-/raw/master/tlbx and compare the
> syntax highlighting for it in vim and emacs.

You don't say what Emacs version this is with, or which parts you think
Emacs is displaying wrong.  Is it the heredoc thing that's wrong, or
something else in addition?

[Message part 2 (image/png, inline)]

[Message part 3 (text/plain, inline)]

It seems like sh-mode isn't interpreting that "@" as the end of the
heredoc.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #13 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 52496 <at> debbugs.gnu.org, Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 14:30:27 +0100

On Dez 19 2021, Lars Ingebrigtsen wrote:

> It seems like sh-mode isn't interpreting that "@" as the end of the
> heredoc.

ELISP> (let ((s " <<- @\n")) (string-match sh-here-doc-open-re s) (match-string 1 s))
"-"
ELISP> (let ((s " <<- EOF\n")) (string-match sh-here-doc-open-re s) (match-string 1 s))
"EOF"

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #16 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 52496 <at> debbugs.gnu.org, Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 14:33:41 +0100

Andreas Schwab <schwab <at> linux-m68k.org> writes:

>> It seems like sh-mode isn't interpreting that "@" as the end of the
>> heredoc.
>
> ELISP> (let ((s " <<- @\n")) (string-match sh-here-doc-open-re s)
> ELISP> (match-string 1 s))
> "-"
> ELISP> (let ((s " <<- EOF\n")) (string-match sh-here-doc-open-re s)
> ELISP> (match-string 1 s))
> "EOF"

Right.  I haven't real the shell documentation -- is "@" a valid heredoc
... separator?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #19 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 52496 <at> debbugs.gnu.org, Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 14:57:56 +0100

On Dez 19 2021, Lars Ingebrigtsen wrote:

> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
> ... separator?

Any word is.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #22 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 52496 <at> debbugs.gnu.org, Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 15:10:22 +0100

Andreas Schwab <schwab <at> linux-m68k.org> writes:

> On Dez 19 2021, Lars Ingebrigtsen wrote:
>
>> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
>> ... separator?
>
> Any word is.

But is @ word constituent?

It's not in sh-mode's word definition, anyway.  The following tweak
fixes this particular problem, but I have no idea whether it was what
what originally reported.

diff --git a/lisp/progmodes/sh-script.el b/lisp/progmodes/sh-script.el
index c6b6f83471..01bebe4a23 100644
--- a/lisp/progmodes/sh-script.el
+++ b/lisp/progmodes/sh-script.el
@@ -867,7 +867,7 @@ sh-here-doc-syntax
     "\\(?:\\(?:.*[^\\\n]\\)?\\(?:\\\\\\\\\\)*\\\\\n\\)*.*")
 
   (defconst sh-here-doc-open-re
-    (concat "[^<]<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\|[-/~._]\\)+\\)"
+    (concat "[^<]<<-?\\s-*\\\\?\\(\\(?:['\"][^'\"]+['\"]\\|\\sw\\|[-/~._@]\\)+\\)"
             sh-escaped-line-re "\\(\n\\)")))
 
 (defun sh--inside-noncommand-expression (pos)


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #27 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 52496 <at> debbugs.gnu.org, Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 15:47:06 +0100

On Dez 19 2021, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab <at> linux-m68k.org> writes:
>
>> On Dez 19 2021, Lars Ingebrigtsen wrote:
>>
>>> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
>>> ... separator?
>>
>> Any word is.
>
> But is @ word constituent?

A shell word can be almost anything.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Message #30 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Kévin Le Gouguec <kevin.legouguec <at> gmail.com>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 52496 <at> debbugs.gnu.org, Lars Ingebrigtsen <larsi <at> gnus.org>,
 Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 16:46:56 +0100

Andreas Schwab <schwab <at> linux-m68k.org> writes:

> On Dez 19 2021, Lars Ingebrigtsen wrote:
>
>> Andreas Schwab <schwab <at> linux-m68k.org> writes:
>>
>>> On Dez 19 2021, Lars Ingebrigtsen wrote:
>>>
>>>> Right.  I haven't real the shell documentation -- is "@" a valid heredoc
>>>> ... separator?
>>>
>>> Any word is.
>>
>> But is @ word constituent?
>
> A shell word can be almost anything.

Relevant part of (bash) Redirections:

>    The format of here-documents is:
>      [N]<<[-]WORD
>              HERE-DOCUMENT
>      DELIMITER
> 
>    No parameter and variable expansion, command substitution, arithmetic
> expansion, or filename expansion is performed on WORD.  If any part of
> WORD is quoted, the DELIMITER is the result of quote removal on WORD,

And (bash) Definitions:

> 'word'
>      A sequence of characters treated as a unit by the shell.  Words may
>      not include unquoted 'metacharacters'.

Preceded by:

> 'metacharacter'
>      A character that, when unquoted, separates words.  A metacharacter
>      is a 'space', 'tab', 'newline', or one of the following characters:
>      '|', '&', ';', '(', ')', '<', or '>'.

(AFAICT the same goes for POSIX sh[1])

So IIUC for here-documents anything is fair game except (1)
{single,double,back} quotes (2) metacharacters (3) backslashes and
comment openers (#); all of which can still be used if quoted or escaped
*on the opening line only*.


[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04

Message #33 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Cc: 52496 <at> debbugs.gnu.org
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Sun, 19 Dec 2021 20:42:26 +0100

Yilkal Argaw <yilkalargawworkneh <at> gmail.com> writes:

> Yes it is the heredoc part. I have tried emacs 27, 28 and 29(i.e. the
> master) and it all behaves the same.

Right.  So the question is whether we should just continue to add
characters to that regexp (safe solution) or whether to try to amend the
shell-script-mode concept of a "word", which I think is likely to break
stuff, but would be more correct...  probably...

Message #36 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Lars Ingebrigtsen <larsi <at> gnus.org>
To: Kévin Le Gouguec <kevin.legouguec <at> gmail.com>
Cc: 52496 <at> debbugs.gnu.org, Andreas Schwab <schwab <at> linux-m68k.org>,
 Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Mon, 20 Dec 2021 10:46:03 +0100

Kévin Le Gouguec <kevin.legouguec <at> gmail.com> writes:

> So IIUC for here-documents anything is fair game except (1)
> {single,double,back} quotes (2) metacharacters (3) backslashes and
> comment openers (#); all of which can still be used if quoted or escaped
> *on the opening line only*.

Thanks.

As far as I can see, sh-mode doesn't really try too hard to be correct
here -- it accepts word constituents, but then adds back -/~._ which it
has defined as punctuation in `sh-mode-syntax-table'.  So perhaps we
should just continue doing that and add @, too, even though that's not
very satisfactory.

So I've now done that in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Message #39 received at 52496 <at> debbugs.gnu.org (full text, mbox):

From: Yilkal Argaw <yilkalargawworkneh <at> gmail.com>
To: Lars Ingebrigtsen <larsi <at> gnus.org>
Cc: 52496 <at> debbugs.gnu.org, Andreas Schwab <schwab <at> linux-m68k.org>,
 Kévin Le Gouguec <kevin.legouguec <at> gmail.com>
Subject: Re: bug#52496: syntax highlighting problem on emacs shell-script mode
Date: Tue, 21 Dec 2021 07:20:47 +0300

Yes, I agree.  I think this is enough for now until a refactor becomes
necessary.

On Mon, Dec 20, 2021 at 12:46 PM Lars Ingebrigtsen <larsi <at> gnus.org> wrote:
>
> Kévin Le Gouguec <kevin.legouguec <at> gmail.com> writes:
>
> > So IIUC for here-documents anything is fair game except (1)
> > {single,double,back} quotes (2) metacharacters (3) backslashes and
> > comment openers (#); all of which can still be used if quoted or escaped
> > *on the opening line only*.
>
> Thanks.
>
> As far as I can see, sh-mode doesn't really try too hard to be correct
> here -- it accepts word constituents, but then adds back -/~._ which it
> has defined as punctuation in `sh-mode-syntax-table'.  So perhaps we
> should just continue doing that and add @, too, even though that's not
> very satisfactory.
>
> So I've now done that in Emacs 29.
>
> --
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.