GNU bug report logs - #79192
json parsing error position reform

Previous Next

To reply to this bug, email your comments to 79192 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Emacs Bug Report <bug-gnu-emacs <at> gnu.org>
Subject: json parsing error position reform
Date: Thu, 7 Aug 2025 17:19:48 +0200

[Message part 1 (text/plain, inline)]

The json parser (src/json.c) spends a surprising amount of time keeping track of the current line and column, on the off chance that an error occurs in which case that information would helpfully be included. This is a bad design in a variety of ways:

(1) It goes against all performance engineering to spend resources on maintaining information that is unlikely to be used.
(2) JSON parse errors aren't expected to occur during normal operation.
(3) From what I can tell, nobody uses the position information of the json-error signal at all.
(4) One reason for (3) is that it isn't documented anywhere.
(5) Even if someone would find the position useful, there is no evidence that they would need the line and column (which could easily be derived from the position).

So let's remove at least the line and column of the error position. We can still supply the position from the start of the parse because we have the information anyway.

There's a patch. Surprisingly, it speeds up the parsing part by more than 10 % which is definitely worthwhile.

[jsonspeed.diff (application/octet-stream, attachment)]

Message #8 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Thu, 07 Aug 2025 18:36:16 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Thu, 7 Aug 2025 17:19:48 +0200
> 
> The json parser (src/json.c) spends a surprising amount of time keeping track of the current line and column, on the off chance that an error occurs in which case that information would helpfully be included. This is a bad design in a variety of ways:
> 
> (1) It goes against all performance engineering to spend resources on maintaining information that is unlikely to be used.
> (2) JSON parse errors aren't expected to occur during normal operation.
> (3) From what I can tell, nobody uses the position information of the json-error signal at all.
> (4) One reason for (3) is that it isn't documented anywhere.
> (5) Even if someone would find the position useful, there is no evidence that they would need the line and column (which could easily be derived from the position).
> 
> So let's remove at least the line and column of the error position. We can still supply the position from the start of the parse because we have the information anyway.

If we have the position, can we have the cake and eat it, too?  Would
it be possible to call some function in the case of an error that
would calculate the line and column from the position, and include
that in the error data?  That way, the performance will suffer only
when there is an error to report.

Another idea is to disable this conditionally, so that if someone
really needs this, they could have it for the price of somewhat slower
parsing.

Message #11 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Thu, 7 Aug 2025 17:43:26 +0200

7 aug. 2025 kl. 17.36 skrev Eli Zaretskii <eliz <at> gnu.org>:

> If we have the position, can we have the cake and eat it, too?  Would
> it be possible to call some function in the case of an error that
> would calculate the line and column from the position, and include
> that in the error data?  That way, the performance will suffer only
> when there is an error to report.

It would be completely useless busy-work in practice. Even if disabled by default, it's programming work done for something that makes nobody happy. Let's put our effort where it counts instead.

Message #14 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Thu, 07 Aug 2025 19:17:44 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Thu, 7 Aug 2025 17:43:26 +0200
> Cc: 79192 <at> debbugs.gnu.org
> 
> 7 aug. 2025 kl. 17.36 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
> > If we have the position, can we have the cake and eat it, too?  Would
> > it be possible to call some function in the case of an error that
> > would calculate the line and column from the position, and include
> > that in the error data?  That way, the performance will suffer only
> > when there is an error to report.
> 
> It would be completely useless busy-work in practice. Even if disabled by default, it's programming work done for something that makes nobody happy. Let's put our effort where it counts instead.

Sorry, I don't follow.  How is providing helpful data about the locus
of an error "useless"?

Message #17 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Thu, 7 Aug 2025 18:32:34 +0200

7 aug. 2025 kl. 18.17 skrev Eli Zaretskii <eliz <at> gnu.org>:

> How is providing helpful data about the locus
> of an error "useless"?

Sorry, I didn't intend to sound facetious. What I meant is that the line and column are not likely to help current or future users in any significant way:

- If anyone would need the point of error, it would almost certainly be as a position, not line and column.
- In the unlikely case that the line and column would be called for, they can easily be computed by the user.
- That computation would be done in Lisp, which is easier; ours would have to be in C.
- Line and column aren't universally well-defined (start at 0 or 1? How are columns defined? Tabs? Multibyte chars? Combining chars?). The user is better-placed to use a definition that suits the requirements.
- But so far, nobody seems to have a need the point of error in the first place.

Therefore it very much looks like gold-plating. In other words, we'd give a mistake in the first implementation an unwarranted life extension when we had a chance to drop it without consequences.

Message #20 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Stéphane Marks <shipmints <at> gmail.com>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: Eli Zaretskii <eliz <at> gnu.org>, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Thu, 7 Aug 2025 13:15:18 -0400

[Message part 1 (text/plain, inline)]

On Thu, Aug 7, 2025 at 12:33 PM Mattias Engdegård <
mattias.engdegard <at> gmail.com> wrote:

> 7 aug. 2025 kl. 18.17 skrev Eli Zaretskii <eliz <at> gnu.org>:
>
> > How is providing helpful data about the locus
> > of an error "useless"?
>
> Sorry, I didn't intend to sound facetious. What I meant is that the line
> and column are not likely to help current or future users in any
> significant way:
>
> - If anyone would need the point of error, it would almost certainly be as
> a position, not line and column.
> - In the unlikely case that the line and column would be called for, they
> can easily be computed by the user.
> - That computation would be done in Lisp, which is easier; ours would have
> to be in C.
> - Line and column aren't universally well-defined (start at 0 or 1? How
> are columns defined? Tabs? Multibyte chars? Combining chars?). The user is
> better-placed to use a definition that suits the requirements.
> - But so far, nobody seems to have a need the point of error in the first
> place.
>
> Therefore it very much looks like gold-plating. In other words, we'd give
> a mistake in the first implementation an unwarranted life extension when we
> had a chance to drop it without consequences.
>

Most of the json I deal with (not in Emacs, though, or yet) is "minified"
and has no newlines.  The position would be all I could reasonably use for
minified json as the line number would always be 1 and the column number
would, I'd hope, equal the position of an error.

[Message part 2 (text/html, inline)]

Message #23 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Thu, 07 Aug 2025 21:22:46 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Thu, 7 Aug 2025 18:32:34 +0200
> Cc: 79192 <at> debbugs.gnu.org
> 
> 7 aug. 2025 kl. 18.17 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
> > How is providing helpful data about the locus
> > of an error "useless"?
> 
> Sorry, I didn't intend to sound facetious. What I meant is that the line and column are not likely to help current or future users in any significant way:
> 
> - If anyone would need the point of error, it would almost certainly be as a position, not line and column.
> - In the unlikely case that the line and column would be called for, they can easily be computed by the user.
> - That computation would be done in Lisp, which is easier; ours would have to be in C.
> - Line and column aren't universally well-defined (start at 0 or 1? How are columns defined? Tabs? Multibyte chars? Combining chars?). The user is better-placed to use a definition that suits the requirements.
> - But so far, nobody seems to have a need the point of error in the first place.
> 
> Therefore it very much looks like gold-plating. In other words, we'd give a mistake in the first implementation an unwarranted life extension when we had a chance to drop it without consequences.

Every tool that reports errors or hits shows line numbers, and
sometimes also column numbers.  So it looks like a reasonable thing to
expect.  Whether they are zero-based or 1-based we could decide by
looking at what other tools do (I think 1-based), but that's not the
important part.

As for calculating lines and columns, I indeed had in mind a call to
some Lisp function, which is done only when we are about to signal an
error.  We don't have to do that in C.

Message #26 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dmitry <at> gutov.dev>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>,
 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Fri, 8 Aug 2025 03:43:35 +0300

On 07/08/2025 18:19, Mattias Engdegård wrote:
> (2) JSON parse errors aren't expected to occur during normal operation.
> (3) From what I can tell, nobody uses the position information of the json-error signal at all.

More than that: none of the packages I have installed (at least) even 
handle that error.

Message #29 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Dmitry Gutov <dmitry <at> gutov.dev>, Ship Mints <shipmints <at> gmail.com>,
 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Sat, 9 Aug 2025 16:08:12 +0200

7 aug. 2025 kl. 20.22 skrev Eli Zaretskii <eliz <at> gnu.org>:

> Every tool that reports errors or hits shows line numbers, and
> sometimes also column numbers.  So it looks like a reasonable thing to
> expect.

But the json parser isn't a tool, it's a low-level primitive that can be used to make tools.
Having it produce line and column numbers makes as much sense as doing that from `re-search-forward`.
Or, if you prefer, from one of our two (three?) XML parser APIs. It just not how APIs are designed.

Message #32 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: dmitry <at> gutov.dev, shipmints <at> gmail.com, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Sat, 09 Aug 2025 17:16:56 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Sat, 9 Aug 2025 16:08:12 +0200
> Cc: 79192 <at> debbugs.gnu.org,
>  Dmitry Gutov <dmitry <at> gutov.dev>,
>  Ship Mints <shipmints <at> gmail.com>
> 
> 7 aug. 2025 kl. 20.22 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
> > Every tool that reports errors or hits shows line numbers, and
> > sometimes also column numbers.  So it looks like a reasonable thing to
> > expect.
> 
> But the json parser isn't a tool, it's a low-level primitive that can be used to make tools.
> Having it produce line and column numbers makes as much sense as doing that from `re-search-forward`.
> Or, if you prefer, from one of our two (three?) XML parser APIs. It just not how APIs are designed.

What are the practical problems of calculating the line number only
when an error is about to be reported?  That shouldn't affect
performance, so I don't see why we would be against doing that.

Message #35 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Dmitry Gutov <dmitry <at> gutov.dev>, Ship Mints <shipmints <at> gmail.com>,
 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Sun, 10 Aug 2025 13:50:00 +0200

9 aug. 2025 kl. 16.16 skrev Eli Zaretskii <eliz <at> gnu.org>:

> What are the practical problems of calculating the line number only
> when an error is about to be reported?  That shouldn't affect
> performance, so I don't see why we would be against doing that.

What are the pressing needs for doing that?

Of the callers of the json parser, not all catch its errors specifically.

Of those that do, most(*) do not use the position information.
(*) currently 100 %

Of those that would use the position information, many wouldn't need line and column numbers.

Want to position the cursor at the point of the parse error? Don't need line or column.
Want to use faces to show what has been successfully parsed and what has not? Don't need line or column.
Want to write a flymake tool that reports the error? Don't need line or column.

We should not compute line and column in the json parser because there's no need, and it wouldn't be good API design. Adding code willy-nilly just because it doesn't slow down normal use much isn't good enough a reason.

Message #38 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: dmitry <at> gutov.dev, shipmints <at> gmail.com, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Sun, 10 Aug 2025 15:03:34 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Sun, 10 Aug 2025 13:50:00 +0200
> Cc: 79192 <at> debbugs.gnu.org,
>  Dmitry Gutov <dmitry <at> gutov.dev>,
>  Ship Mints <shipmints <at> gmail.com>
> 
> 9 aug. 2025 kl. 16.16 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
> > What are the practical problems of calculating the line number only
> > when an error is about to be reported?  That shouldn't affect
> > performance, so I don't see why we would be against doing that.
> 
> What are the pressing needs for doing that?

That we produce this information now.  Removing it would be a breaking
change, and I'd like to avoid that.

> Of the callers of the json parser, not all catch its errors specifically.
> 
> Of those that do, most(*) do not use the position information.
> (*) currently 100 %
> 
> Of those that would use the position information, many wouldn't need line and column numbers.
> 
> Want to position the cursor at the point of the parse error? Don't need line or column.
> Want to use faces to show what has been successfully parsed and what has not? Don't need line or column.
> Want to write a flymake tool that reports the error? Don't need line or column.
> 
> We should not compute line and column in the json parser because there's no need, and it wouldn't be good API design. Adding code willy-nilly just because it doesn't slow down normal use much isn't good enough a reason.

I'm against incompatible changes if we can avoid that.  Our abilities
to be sure that no one uses some feature are very limited and fail
time and again.  So if we can keep this feature without the
performance hit, I'd very much prefer it.

Message #41 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Dmitry Gutov <dmitry <at> gutov.dev>, shipmints <at> gmail.com, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Mon, 11 Aug 2025 14:22:01 +0200

10 aug. 2025 kl. 14.03 skrev Eli Zaretskii <eliz <at> gnu.org>:

> I'm against incompatible changes if we can avoid that.  Our abilities
> to be sure that no one uses some feature are very limited and fail
> time and again.  So if we can keep this feature without the
> performance hit, I'd very much prefer it.


Yes, that's clearly a valid concern and I agree with you in principle. When I started working on this I assumed that I'd have to include the line and column calculations because they were there before.

And while it's theoretically conceivable that there might be a use in a file on someone's private disk somewhere, the fact that no public use could be found at all, that it was never documented, and that it had only been available for a comparatively short time (since the release of Emacs 30), all taken together makes it very unlikely that anything would depend on it.

So it seems to be an opportunity to fix a mistake before being too late. I remember asking the original author of the json parser about it at the time and got the impression that he'd found it useful at some point while debugging the parser itself, but it really didn't make much sense and I made a mental note to get rid of it before the release of Emacs 30. Goes to show how reliable my mental notes can be.

As other people pointed out, the primary use of the json parser is to decode data generated by other programs and these cram together everything without whitespace for bandwidth efficiency, like our own json serialiser does.

A minor point: the parser also reports the error position (the third error parameter, after the line and column) in characters even when the input was a unibyte string or buffer. That doesn't seem right either and would refer to the wrong position in the string or buffer so I'm fixing that bug while at it.

Message #44 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: dmitry <at> gutov.dev, shipmints <at> gmail.com, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Mon, 11 Aug 2025 16:40:40 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Mon, 11 Aug 2025 14:22:01 +0200
> Cc: 79192 <at> debbugs.gnu.org,
>  Dmitry Gutov <dmitry <at> gutov.dev>,
>  shipmints <at> gmail.com
> 
> 10 aug. 2025 kl. 14.03 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
> > I'm against incompatible changes if we can avoid that.  Our abilities
> > to be sure that no one uses some feature are very limited and fail
> > time and again.  So if we can keep this feature without the
> > performance hit, I'd very much prefer it.
> 
> 
> Yes, that's clearly a valid concern and I agree with you in principle. When I started working on this I assumed that I'd have to include the line and column calculations because they were there before.
> 
> And while it's theoretically conceivable that there might be a use in a file on someone's private disk somewhere, the fact that no public use could be found at all, that it was never documented, and that it had only been available for a comparatively short time (since the release of Emacs 30), all taken together makes it very unlikely that anything would depend on it.
> 
> So it seems to be an opportunity to fix a mistake before being too late. I remember asking the original author of the json parser about it at the time and got the impression that he'd found it useful at some point while debugging the parser itself, but it really didn't make much sense and I made a mental note to get rid of it before the release of Emacs 30. Goes to show how reliable my mental notes can be.
> 
> As other people pointed out, the primary use of the json parser is to decode data generated by other programs and these cram together everything without whitespace for bandwidth efficiency, like our own json serialiser does.
> 
> A minor point: the parser also reports the error position (the third error parameter, after the line and column) in characters even when the input was a unibyte string or buffer. That doesn't seem right either and would refer to the wrong position in the string or buffer so I'm fixing that bug while at it.

I hear you, but then counting lines in Lisp is simple, and we have
functions ready to do that.  So how hard is it to call one of those
functions from the parser, when it detects and error and is about to
signal?

Message #47 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: dmitry <at> gutov.dev, shipmints <at> gmail.com, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Tue, 12 Aug 2025 19:53:42 +0200

11 aug. 2025 kl. 15.40 skrev Eli Zaretskii <eliz <at> gnu.org>:

> I hear you, but then counting lines in Lisp is simple, and we have
> functions ready to do that.  So how hard is it to call one of those
> functions from the parser, when it detects and error and is about to
> signal?

The user doesn't care what calculations we make. The values are unused. (And from the user's perspective, undefined.)

Message #50 received at 79192 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Mattias Engdegård <mattias.engdegard <at> gmail.com>
Cc: dmitry <at> gutov.dev, shipmints <at> gmail.com, 79192 <at> debbugs.gnu.org
Subject: Re: bug#79192: json parsing error position reform
Date: Tue, 12 Aug 2025 21:12:54 +0300

> From: Mattias Engdegård <mattias.engdegard <at> gmail.com>
> Date: Tue, 12 Aug 2025 19:53:42 +0200
> Cc: 79192 <at> debbugs.gnu.org,
>  dmitry <at> gutov.dev,
>  shipmints <at> gmail.com
> 
> 11 aug. 2025 kl. 15.40 skrev Eli Zaretskii <eliz <at> gnu.org>:
> 
> > I hear you, but then counting lines in Lisp is simple, and we have
> > functions ready to do that.  So how hard is it to call one of those
> > functions from the parser, when it detects and error and is about to
> > signal?
> 
> The user doesn't care what calculations we make. The values are unused. (And from the user's perspective, undefined.)

If the values are available, whether to use them or not is up to the
callers, I think.  It is none of our business if they don't use them.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.