GNU bug report logs - #78872
Byte compiler: warning message lacks line and column numbers.

Previous Next

To reply to this bug, email your comments to 78872 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: bug-gnu-emacs <at> gnu.org
Subject: Byte compiler: warning message lacks line and column numbers.
Date: Mon, 23 Jun 2025 12:25:12 +0000

[Message part 1 (text/plain, inline)]

Hello, Mattias and Emacs.

On the master branch at this commit:

commit 1a8e8203a158f6bdc74e38f2cdb3214b6c06b9d8 (HEAD -> master,
origin/master, origin/HEAD)
Author: Alan Mackenzie <acm <at> muc.de>
Date:   Fri Jun 20 21:13:45 2025 +0000

    CC Mode: Remove workarounds for missing "\_<", "\_>" in XEmacs

..

With the attached file, at the command line in GNU/Linux, do:

$ cd .../path/to/emacs/lisp/progmodes
$ <Extract test-cc-awk.el to this directory.>
$ .../path/to/emacs/src/emacs -Q -batch -f batch-byte-compile test-cc-awk.el

..  This gives four warning messages, three of which are just artifacts
of the extraction of the test file.

The fourth looks like:

test-cc-awk.el: Warning: ‘font-lock-keyword-face’ is an obsolete variable (as of 31.1); use the quoted symbol instead: 'font-lock-keyword-face

..  This message has no line and column numbers indicating where the
warning was detected.  This is a bug.

Mattias, could you look at this, please.

Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

[test-cc-awk.el (text/plain, attachment)]

Message #8 received at 78872 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: 78872 <at> debbugs.gnu.org
Cc: Mattias Engdegård <mattiase <at> acm.org>, acm <at> muc.de
Subject: Re: bug#78872: Byte compiler: warning message lacks line and column
 numbers.
Date: Fri, 27 Jun 2025 13:56:58 +0000

Hello, also Stefan.

On Mon, Jun 23, 2025 at 12:25:12 +0000, Alan Mackenzie wrote:
> Hello, Mattias and Emacs.

> On the master branch at this commit:

> commit 1a8e8203a158f6bdc74e38f2cdb3214b6c06b9d8 (HEAD -> master,
> origin/master, origin/HEAD)
> Author: Alan Mackenzie <acm <at> muc.de>
> Date:   Fri Jun 20 21:13:45 2025 +0000

>     CC Mode: Remove workarounds for missing "\_<", "\_>" in XEmacs

> ..

> With the attached file, at the command line in GNU/Linux, do:

> $ cd .../path/to/emacs/lisp/progmodes
> $ <Extract test-cc-awk.el to this directory.>
> $ .../path/to/emacs/src/emacs -Q -batch -f batch-byte-compile test-cc-awk.el

> ..  This gives four warning messages, three of which are just artifacts
> of the extraction of the test file.

> The fourth looks like:

> test-cc-awk.el: Warning: ‘font-lock-keyword-face’ is an obsolete variable (as of 31.1); use the quoted symbol instead: 'font-lock-keyword-face

> ..  This message has no line and column numbers indicating where the
> warning was detected.  This is a bug.

> Mattias, could you look at this, please.

> Thanks!

I've bisected the repository to find the place where this bug arose, and
it turns out it was the fix to

  bug#58601: 29.0.50; Infinite loop in byte-compile--first-symbol-with-pos

where a limit of 10 to the search depths in byte-compile-form-stack was
imposed to prevent an infinite loop in rare circumstances.  This was in
function byte-compile--first-symbol-with-pos.

It turns out 10 is far too small, the smallest integer for which a
position appears for test-cc-awk.el being 14.

I seems to me that counting the depth of search is here sub-optimal.
The necessary maximum depth for a deeply nested form would exceed any
reasonable value we could come up with.  I suggest that the function be
reformulated to record conses in a hash table to detect any loops; this
has been done successfully for other lists where we need to detect
loops.

I will implement this in the next few days if I don't hear objections
from others.

Incidentally, the position in the warning message which gets output is
wrong.  I will address this in another post or bug report.

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #11 received at 78872 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Alan Mackenzie <acm <at> muc.de>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 78872 <at> debbugs.gnu.org
Subject: Re: bug#78872: Byte compiler: warning message lacks line and column
 numbers.
Date: Sat, 28 Jun 2025 15:35:04 -0400

Hi Alan,

>> The fourth looks like:
>> test-cc-awk.el: Warning: ‘font-lock-keyword-face’ is an obsolete variable
>> (as of 31.1); use the quoted symbol instead: 'font-lock-keyword-face
[...]
> where a limit of 10 to the search depths in byte-compile-form-stack was
> imposed to prevent an infinite loop in rare circumstances.  This was in
> function byte-compile--first-symbol-with-pos.
>
> It turns out 10 is far too small, the smallest integer for which a
> position appears for test-cc-awk.el being 14.

IIRC we look for position info first in the "relevant sexp" (the sexp
passed to `byte-compile--first-symbol-with-pos`) and if that fails, then
we fallback on the position info from the context
(i.e. `byte-compile-form-stack`).

Why is it that the fallback doesn't work here?

[ I don't understand where the above warning comes from, so I'm probably
  missing something.  AFAICT the only place where
  `font-lock-keyword-face` appears is within a `quote` so there's no
  chance it will elicit such a warning (although it also deserves such
  a warning, admittedly 🙂).  ]

> I seems to me that counting the depth of search is here sub-optimal.

No doubt.  I thought it was OK because of the fallback to
`byte-compile-form-stack` and also because returning nil occasionally is
just a fact of life: the form may fail to contain any sympos, after all.

> The necessary maximum depth for a deeply nested form would exceed any
> reasonable value we could come up with.  I suggest that the function be
> reformulated to record conses in a hash table to detect any loops; this
> has been done successfully for other lists where we need to detect
> loops.

No objection on my side.

BTW, I get the impression that the larger the term (and thus, the harder
we work to find a sympos inside of it), the higher the risk of that
sympos not being The Right One.  It's probably not a big deal either
way, tho, since we just don't have enough info to determine which sympos
is The Right One anyway.

> Incidentally, the position in the warning message which gets output is
> wrong.  I will address this in another post or bug report.

I promise I wrote the above before reading this part.  🙂


        Stefan

Message #14 received at 78872 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 78872 <at> debbugs.gnu.org
Subject: Re: bug#78872: Byte compiler: warning message lacks line and column
 numbers.
Date: Sun, 29 Jun 2025 20:45:21 +0000

Hello, Stefan.

On Sat, Jun 28, 2025 at 15:35:04 -0400, Stefan Monnier wrote:
> Hi Alan,

> >> The fourth looks like:
> >> test-cc-awk.el: Warning: ‘font-lock-keyword-face’ is an obsolete variable
> >> (as of 31.1); use the quoted symbol instead: 'font-lock-keyword-face
> [...]
> > where a limit of 10 to the search depths in byte-compile-form-stack was
> > imposed to prevent an infinite loop in rare circumstances.  This was in
> > function byte-compile--first-symbol-with-pos.

> > It turns out 10 is far too small, the smallest integer for which a
> > position appears for test-cc-awk.el being 14.

> IIRC we look for position info first in the "relevant sexp" (the sexp
> passed to `byte-compile--first-symbol-with-pos`) and if that fails, then
> we fallback on the position info from the context
> (i.e. `byte-compile-form-stack`).

> Why is it that the fallback doesn't work here?

I think it's because all the symbols up to a car/cdr/combination of 10
are lacking positions.  These symbols are generated by the
compiler or macro expander (more the latter, I think).

> [ I don't understand where the above warning comes from, so I'm probably
>   missing something.  AFAICT the only place where
>   `font-lock-keyword-face` appears is within a `quote` so there's no
>   chance it will elicit such a warning (although it also deserves such
>   a warning, admittedly 🙂).  ]

By the time '(1 font-lock-keyword-place t) gets used by the byte-compile
inside c-make-font-lock-search-function, that quote has been evaluated
away.

> > I seems to me that counting the depth of search is here sub-optimal.

> No doubt.  I thought it was OK because of the fallback to
> `byte-compile-form-stack` and also because returning nil occasionally is
> just a fact of life: the form may fail to contain any sympos, after all.

But it happens for a very shallowly nested form, here.

> > The necessary maximum depth for a deeply nested form would exceed any
> > reasonable value we could come up with.  I suggest that the function be
> > reformulated to record conses in a hash table to detect any loops; this
> > has been done successfully for other lists where we need to detect
> > loops.

> No objection on my side.

I will do this work, soon.

> BTW, I get the impression that the larger the term (and thus, the harder
> we work to find a sympos inside of it), the higher the risk of that
> sympos not being The Right One.  It's probably not a big deal either
> way, tho, since we just don't have enough info to determine which sympos
> is The Right One anyway.

> > Incidentally, the position in the warning message which gets output is
> > wrong.  I will address this in another post or bug report.

> I promise I wrote the above before reading this part.  🙂

I've spent the weekend working out why that happens.  The position given
for the unquoted font-lock-keywords-face in

(defconst vr-foo
  (eval-when-compile
    `(
      (,(c-make-font-lock-search-function
         "^\\s *\\(func\\(tion\\)?\\)\\s +\\(\\(\\sw+\\(::\\sw+\\)?\\)\\s *\\)?\\(([^()]*)\\)?"
         '(1 font-lock-keyword-face t))))))

is that of the eval-when-compile symbol.  This is obviously bad.

c-make-font-lock-search-function contains an explicit byte-compile form,
and I think it is this that produces the warning message.

What's happening is that eval-when-compile is wrongly compiling its
argument, then evaluating it.  This compilation removes the symbols'
positions, so they are no longer available when
c-make-font-lock-search-function's byte-compile runs.

eval-when-compile should NOT compile its argument.  It should evaluate
it.  When I remove that compilation, I get the correct position for the
warning message about font-lock-keyword-face.

Interestingly enough, eval-AND-compile _doesn't_ compile its argument.
It just macro-expands it, then both evaluates the result of the
expansion, and returns it.  Presumably the caller ends up compiling it,
I'm not sure.

I think I will post on emacs-devel about fixing that obtrusive
compilation in eval-when-compile.  Other developers may have opinions
about it.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #17 received at 78872 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Alan Mackenzie <acm <at> muc.de>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 78872 <at> debbugs.gnu.org
Subject: Re: bug#78872: Byte compiler: warning message lacks line and column
 numbers.
Date: Mon, 30 Jun 2025 00:17:35 -0400

>> Why is it that the fallback doesn't work here?
> I think it's because all the symbols up to a car/cdr/combination of 10
> are lacking positions.  These symbols are generated by the
> compiler or macro expander (more the latter, I think).

See below.

>> [ I don't understand where the above warning comes from, so I'm probably
>>   missing something.  AFAICT the only place where
>>   `font-lock-keyword-face` appears is within a `quote` so there's no
>>   chance it will elicit such a warning (although it also deserves such
>>   a warning, admittedly 🙂).  ]
>
> By the time '(1 font-lock-keyword-place t) gets used by the byte-compile
> inside c-make-font-lock-search-function, that quote has been evaluated
> away.

Aaahhhh!

I think that's the culprit: the code with the `font-lock-keyword-face`
reference is generated by a function and passed to `byte-compile`.
There can't be any sympos in it for that reason (we need to strip
sympos before we pass any sexp to `eval`).

If we want good warnings with good location information, we need to
replace the `c-make-font-lock-search-function` function (which
internally calls `byte-compile`) with a macro (which doesn't call
`byte-compile` but instead returns the corresponding code so it gets
byte-compiled by "the outer compiler").
[ I have a trivial patch to do that, but sadly it doesn't quite fit in
  the margin so I'll leave it as an exercise for the reader.  ]

>> > I seems to me that counting the depth of search is here sub-optimal.
>> No doubt.  I thought it was OK because of the fallback to
>> `byte-compile-form-stack` and also because returning nil occasionally is
>> just a fact of life: the form may fail to contain any sympos, after all.
> But it happens for a very shallowly nested form, here.

Actually, I think it happens here because the stack is empty in the
inner `byte-compile` (it would have to consult the stack of the outer
compiler, but the two compiler invocations don't know each other, so
it's not that simple).

> What's happening is that eval-when-compile is wrongly compiling its
> argument, then evaluating it.  This compilation removes the symbols'
> positions, so they are no longer available when
> c-make-font-lock-search-function's byte-compile runs.

The symbols' positions have to be removed before we can run the code,
regardless if we compile the code or not, otherwise sympos can "leak"
outside their compilation.

> Interestingly enough, eval-AND-compile _doesn't_ compile its argument.
> It just macro-expands it, then both evaluates the result of the
> expansion, and returns it.

But the inner `byte-compile` would give a similarly poor location info
in the warning because `eval-and-compile` also strips the sympos before
evaluating the code.


        Stefan

Message #20 received at 78872 <at> debbugs.gnu.org (full text, mbox):

From: Alan Mackenzie <acm <at> muc.de>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 78872 <at> debbugs.gnu.org, acm <at> muc.de
Subject: Re: bug#78872: Byte compiler: warning message lacks line and column
 numbers.
Date: Mon, 30 Jun 2025 11:29:19 +0000

Hello, Stefan.

Thanks for such a rapid and helpful reply.

On Mon, Jun 30, 2025 at 00:17:35 -0400, Stefan Monnier wrote:
> >> Why is it that the fallback doesn't work here?
> > I think it's because all the symbols up to a car/cdr/combination of 10
> > are lacking positions.  These symbols are generated by the
> > compiler or macro expander (more the latter, I think).

> See below.

> >> [ I don't understand where the above warning comes from, so I'm probably
> >>   missing something.  AFAICT the only place where
> >>   `font-lock-keyword-face` appears is within a `quote` so there's no
> >>   chance it will elicit such a warning (although it also deserves such
> >>   a warning, admittedly 🙂).  ]

> > By the time '(1 font-lock-keyword-place t) gets used by the byte-compile
> > inside c-make-font-lock-search-function, that quote has been evaluated
> > away.

> Aaahhhh!

The warning is correct, though.  That form needs to look like
    '(1 'font-lock-keyword-place t)
, and when it's written like that it works fine, using the symbol rather
than the obsolete variable for the face.

> I think that's the culprit: the code with the `font-lock-keyword-face`
> reference is generated by a function and passed to `byte-compile`.
> There can't be any sympos in it for that reason (we need to strip
> sympos before we pass any sexp to `eval`).

The need to strip SWPs is solely to prevent their leakage, as you
mention.  Emacs bootstraps quite happily without this stripping, though
there is surely some use of eval-when-compile where it wouldn't be so
happy.

> If we want good warnings with good location information, we need to
> replace the `c-make-font-lock-search-function` function (which
> internally calls `byte-compile`) with a macro (which doesn't call
> `byte-compile` but instead returns the corresponding code so it gets
> byte-compiled by "the outer compiler").

Yes, this would be the best way to go.

> [ I have a trivial patch to do that, but sadly it doesn't quite fit in
>   the margin so I'll leave it as an exercise for the reader.  ]

Your margin is narrow indeed.  I think all that would be needed would be
to replace defun with defmacro, and remove the (byte-compile ...) around
the bulk of the code.  But I haven't tried it out, yet.

> >> > I seems to me that counting the depth of search is here sub-optimal.
> >> No doubt.  I thought it was OK because of the fallback to
> >> `byte-compile-form-stack` and also because returning nil occasionally is
> >> just a fact of life: the form may fail to contain any sympos, after all.
> > But it happens for a very shallowly nested form, here.

> Actually, I think it happens here because the stack is empty in the
> inner `byte-compile` (it would have to consult the stack of the outer
> compiler, but the two compiler invocations don't know each other, so
> it's not that simple).

If I increase the depth limit of 10 to 14 or more, I get the position in
the warning - just the wrong position.  ;-(  I'll need to look into this
a bit more.

> > What's happening is that eval-when-compile is wrongly compiling its
> > argument, then evaluating it.  This compilation removes the symbols'
> > positions, so they are no longer available when
> > c-make-font-lock-search-function's byte-compile runs.

> The symbols' positions have to be removed before we can run the code,
> regardless if we compile the code or not, otherwise sympos can "leak"
> outside their compilation.

Yes, this is the bit I missed yesterday.

> > Interestingly enough, eval-AND-compile _doesn't_ compile its argument.
> > It just macro-expands it, then both evaluates the result of the
> > expansion, and returns it.

> But the inner `byte-compile` would give a similarly poor location info
> in the warning because `eval-and-compile` also strips the sympos before
> evaluating the code.

Yes, it strips the SWPs before passing the code to eval, but leaves them
on for the "-and-compile" bit.

So perhaps the best thing to do is add a warning to the documentation of
byte-compile not to use it inside eval-when/and-compile, but instead to
prefer a macro.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Message #23 received at 78872 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Alan Mackenzie <acm <at> muc.de>
Cc: Mattias Engdegård <mattiase <at> acm.org>,
 78872 <at> debbugs.gnu.org
Subject: Re: bug#78872: Byte compiler: warning message lacks line and column
 numbers.
Date: Mon, 30 Jun 2025 10:55:01 -0400

>> > By the time '(1 font-lock-keyword-place t) gets used by the
>> > byte-compile inside c-make-font-lock-search-function, that quote
>> > has been evaluated away.
>
>> Aaahhhh!
>
> The warning is correct, though.  That form needs to look like
>     '(1 'font-lock-keyword-place t)

+1 !

>> Actually, I think it happens here because the stack is empty in the
>> inner `byte-compile` (it would have to consult the stack of the outer
>> compiler, but the two compiler invocations don't know each other, so
>> it's not that simple).
> If I increase the depth limit of 10 to 14 or more, I get the position
> in the warning - just the wrong position.  ;-(  I'll need to look into
> this a bit more.

I think it's simply that contrary to my belief, `bytecomp.el` doesn't
empty the stack before it starts its work, so the "depth 14" reaches to
the part of the stack that belongs to the outer compilation.

In this case, it happens to be the right thing to do (the position is
not ideal but given that we stripped the sympos it's the best we have),
although in general I think this can lead to confusing location info
(e.g. if the compilation of file A happens to (auto)load file B which
calls `byte-compile` somewhere at top-level, warnings from B could be
misattributed to A).

> So perhaps the best thing to do is add a warning to the documentation
> of byte-compile not to use it inside eval-when/and-compile, but
> instead to prefer a macro.

Fine by me, tho according to `grep` this is not a common "mistake",
so I'm not sure it's worth the trouble.


        Stefan

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.