GNU bug report logs - #21616
[PATCH] Enable sorting of JSON object keys when encoding

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 21616 in the body.
You can then email your comments to 21616 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simen Heggestøyl <simenheg <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: [PATCH] Enable sorting of JSON object keys when encoding
Date: Sun, 04 Oct 2015 21:15:21 +0200

When working with JSON data, it is often convenient to be able to
prettify the data, and having object keys sorted in a fixed order.
This patch adds the variable `json-encoding-object-sort-key' which
enables defining a sort key to be used when encoding JSON objects.
Additionally, the commands `json-pretty-print-ordered' and
`json-pretty-print-buffer-ordered' are added for convenience,
providing prettification with alphabetical ordering of object keys.

This gets rid of a lot of redundant code from `json-encode-hash-table'
and `json-encode-plist' by going via alists, and using the logic of
`json-encode-alist' commonly for all the structures.

I was in doubt whether to make `json-pretty-print-ordered' and
`json-pretty-print-buffer-ordered' their own commands, or if it would
be better to provide this functionality by having `json-pretty-print'
and `json-pretty-print-buffer' accept prefix arguments. I decided on
the former, because I think it makes the commands easier to discover.

A proposed patch follows!

-- Simen


From bf32702520b17e9bb975c68adb79c0d84e2cbf41 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Simen=20Heggest=C3=B8yl?= <simenheg <at> gmail.com>
Date: Sun, 4 Oct 2015 12:21:57 +0200
Subject: [PATCH] Enable sorting of JSON object keys when encoding

* lisp/json.el (json-encoding-object-sort-key): New variable for
specifying a sort key for JSON objects during encoding.
(json--plist-to-alist): New utility function.
(json-encode-hash-table): Re-use `json-encode-alist'.
(json-encode-alist): Sort output by `json-encoding-object-sort-key, when
set.
(json-encode-plist): Re-use `json-encode-alist'.
(json-pretty-print-buffer-ordered): New command to pretty print the
buffer with object keys sorted alphabetically.
(json-pretty-print-ordered): New command to pretty print the region with
object keys sorted alphabetically.

* test/automated/json-tests.el (test-json-plist-to-alist)
(test-json-encode-plist, test-json-encode-hash-table)
(test-json-encode-with-sort-key): New tests.

* etc/NEWS: Add an entry for the new commands.
---
 etc/NEWS                     |  4 +++
 lisp/json.el                 | 77 +++++++++++++++++++++-----------------------
 test/automated/json-tests.el | 22 +++++++++++++
 3 files changed, 63 insertions(+), 40 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index dbe0de3..062f17a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -318,6 +318,10 @@ standards.
 ---
 *** `json-pretty-print' and `json-pretty-print-buffer' now maintain
 the ordering of object keys by default.
+---
+*** New commands `json-pretty-print-ordered' and
+`json-pretty-print-buffer-ordered' pretty prints JSON objects with
+object keys sorted alphabetically.
 
 ** You can recompute the VC state of a file buffer with `M-x vc-refresh-state'
 ** Prog mode has some support for multi-mode indentation.
diff --git a/lisp/json.el b/lisp/json.el
index e2c7cc7..33247bf 100644
--- a/lisp/json.el
+++ b/lisp/json.el
@@ -52,6 +52,8 @@
 
 ;;; Code:
 
+(require 'map)
+
 ;; Parameters
 
 (defvar json-object-type 'alist
@@ -100,6 +102,13 @@ this around your call to `json-read' instead of `setq'ing it.")
   "The default indentation level for encoding.
 Used only when `json-encoding-pretty-print' is non-nil.")
 
+(defvar json-encoding-object-sort-key nil
+  "Sort key for JSON object keys during encoding.
+If nil, no sorting is performed.  Else, JSON object keys are
+ordered by the specified sort key during encoding.  For instance,
+setting this to `string<' will have JSON object keys ordered
+alphabetically.")
+
 (defvar json--encoding-current-indentation "\n"
   "Internally used to keep track of the current indentation level of encoding.
 Used only when `json-encoding-pretty-print' is non-nil.")
@@ -148,6 +157,15 @@ Unlike `reverse', this keeps the property-value pairs intact."
         (push prop res)))
     res))
 
+(defun json--plist-to-alist (plist)
+  "Return an alist of the property-value pairs in PLIST."
+  (let (res)
+    (while plist
+      (let ((prop (pop plist))
+            (val (pop plist)))
+        (push (cons prop val) res)))
+    (nreverse res)))
+
 (defmacro json--with-indentation (body)
   `(let ((json--encoding-current-indentation
           (if json-encoding-pretty-print
@@ -421,32 +439,17 @@ Please see the documentation of `json-object-type' and `json-key-type'."
 
 (defun json-encode-hash-table (hash-table)
   "Return a JSON representation of HASH-TABLE."
-  (format "{%s%s}"
-          (json-join
-           (let (r)
-             (json--with-indentation
-              (maphash
-               (lambda (k v)
-                 (push (format
-                        (if json-encoding-pretty-print
-                            "%s%s: %s"
-                          "%s%s:%s")
-                        json--encoding-current-indentation
-                        (json-encode-key k)
-                        (json-encode v))
-                       r))
-               hash-table))
-             r)
-           json-encoding-separator)
-          (if (or (not json-encoding-pretty-print)
-                  json-encoding-lisp-style-closings)
-              ""
-            json--encoding-current-indentation)))
+  (json-encode-alist (map-into hash-table 'list)))
 
 ;; List encoding (including alists and plists)
 
 (defun json-encode-alist (alist)
   "Return a JSON representation of ALIST."
+  (when json-encoding-object-sort-key
+    (setq alist
+          (sort alist (lambda (a b)
+                        (funcall json-encoding-object-sort-key
+                                 (car a) (car b))))))
   (format "{%s%s}"
           (json-join
            (json--with-indentation
@@ -466,25 +469,7 @@ Please see the documentation of `json-object-type' and `json-key-type'."
 
 (defun json-encode-plist (plist)
   "Return a JSON representation of PLIST."
-  (let (result)
-    (json--with-indentation
-      (while plist
-        (push (concat
-               json--encoding-current-indentation
-               (json-encode-key (car plist))
-               (if json-encoding-pretty-print
-                   ": "
-                 ":")
-               (json-encode (cadr plist)))
-              result)
-        (setq plist (cddr plist))))
-    (concat "{"
-            (json-join (nreverse result) json-encoding-separator)
-            (if (and json-encoding-pretty-print
-                     (not json-encoding-lisp-style-closings))
-                json--encoding-current-indentation
-              "")
-            "}")))
+  (json-encode-alist (json--plist-to-alist plist)))
 
 (defun json-encode-list (list)
   "Return a JSON representation of LIST.
@@ -622,6 +607,18 @@ Advances point just past JSON object."
           (txt (delete-and-extract-region begin end)))
       (insert (json-encode (json-read-from-string txt))))))
 
+(defun json-pretty-print-buffer-ordered ()
+  "Pretty-print current buffer with object keys ordered."
+  (interactive)
+  (let ((json-encoding-object-sort-key 'string<))
+    (json-pretty-print-buffer)))
+
+(defun json-pretty-print-ordered (begin end)
+  "Pretty-print the region with object keys ordered."
+  (interactive "r")
+  (let ((json-encoding-object-sort-key 'string<))
+    (json-pretty-print begin end)))
+
 (provide 'json)
 
 ;;; json.el ends here
diff --git a/test/automated/json-tests.el b/test/automated/json-tests.el
index d1b7a2f..cdf85cc 100644
--- a/test/automated/json-tests.el
+++ b/test/automated/json-tests.el
@@ -28,11 +28,33 @@
   (should (equal (json--plist-reverse '(:a 1 :b 2 :c 3))
                  '(:c 3 :b 2 :a 1))))
 
+(ert-deftest test-json-plist-to-alist ()
+  (should (equal (json--plist-to-alist '()) '()))
+  (should (equal (json--plist-to-alist '(:a 1)) '((:a . 1))))
+  (should (equal (json--plist-to-alist '(:a 1 :b 2 :c 3))
+                 '((:a . 1) (:b . 2) (:c . 3)))))
+
+(ert-deftest test-json-encode-plist ()
+  (let ((plist '(:a 1 :b 2)))
+    (should (equal (json-encode plist) "{\"a\":1,\"b\":2}"))))
+
 (ert-deftest json-encode-simple-alist ()
   (should (equal (json-encode '((a . 1)
                                 (b . 2)))
                  "{\"a\":1,\"b\":2}")))
 
+(ert-deftest test-json-encode-hash-table ()
+  (let ((hash-table (make-hash-table))
+        (json-encoding-object-sort-key 'string<))
+    (puthash :a 1 hash-table)
+    (puthash :b 2 hash-table)
+    (should (equal (json-encode hash-table) "{\"a\":1,\"b\":2}"))))
+
+(ert-deftest test-json-encode-with-sort-key ()
+  (let ((alist '((:a . 1) (:b . 2) (:c . 3)))
+        (json-encoding-object-sort-key 'string>))
+    (should (equal (json-encode alist) "{\"c\":3,\"b\":2,\"a\":1}"))))
+
 (ert-deftest json-read-simple-alist ()
   (let ((json-object-type 'alist))
     (should (equal (json-read-from-string "{\"a\": 1, \"b\": 2}")
-- 
2.5.3

Message #8 received at 21616 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Simen Heggestøyl <simenheg <at> gmail.com>,
 21616 <at> debbugs.gnu.org
Subject: Re: bug#21616: [PATCH] Enable sorting of JSON object keys when
 encoding
Date: Mon, 9 Nov 2015 02:01:25 +0200

Hi Simen,

On 10/04/2015 10:15 PM, Simen Heggestøyl wrote:
> When working with JSON data, it is often convenient to be able to
> prettify the data, and having object keys sorted in a fixed order.
> This patch adds the variable `json-encoding-object-sort-key' which

I think it's a "predicate", not a "key". See the argument names in 
`sort' and `cl-sort': the latter has a :key keyword argument, but it has 
different purpose.

Call it json-[encode-]key-sort-predicate, maybe?

> This gets rid of a lot of redundant code from `json-encode-hash-table'
> and `json-encode-plist' by going via alists, and using the logic of
> `json-encode-alist' commonly for all the structures.

The unification part makes me concerned, again, from the performance 
standpoint. Did you do any measuring here?

> I was in doubt whether to make `json-pretty-print-ordered' and
> `json-pretty-print-buffer-ordered' their own commands, or if it would
> be better to provide this functionality by having `json-pretty-print'
> and `json-pretty-print-buffer' accept prefix arguments. I decided on
> the former, because I think it makes the commands easier to discover.

Yes, making them separate seems to be more in line with other Emacs 
commands.

Message #11 received at 21616 <at> debbugs.gnu.org (full text, mbox):

From: Simen Heggestøyl <simenheg <at> gmail.com>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 21616 <at> debbugs.gnu.org
Subject: Re: bug#21616: [PATCH] Enable sorting of JSON object keys when
 encoding
Date: Wed, 11 Nov 2015 19:59:33 +0100

[Message part 1 (text/plain, inline)]

Hello, Dmitry.

On Mon, Nov 9, 2015 at 1:01 AM, Dmitry Gutov <dgutov <at> yandex.ru> wrote:
> Call it json-[encode-]key-sort-predicate, maybe?

Indeed, "predicate" seems more appropriate. I'll change the name.

> The unification part makes me concerned, again, from the performance 
> standpoint. Did you do any measuring here?

I didn't before now, but as you feared, the performance became
measurably worse when encoding plists and hash-tables.

I performed some benchmarks on the same 560K JSON file as before
(http://folk.uio.no/simenheg/huge.json), with the following results:

With json-object-type set to hash-table:

 Before the patch:
 (benchmark-run 100 (json-encode huge))
      ⇒ (17.396825399 800 7.954568797999999)

 After the patch:
 (benchmark-run 100 (json-encode huge))
      ⇒ (19.338006244000002 700 9.664285849000013)

With json-object-type set to plist:

 Before the patch:
 (benchmark-run 100 (json-encode huge))
      ⇒ (15.152158676 1101 7.109026906)

 After the patch:
 (benchmark-run 100 (json-encode huge))
      ⇒ (18.122579887 777 8.46547749400001)

How about keeping the old encoding code as the default, and only do the
{hash-table, plist} → alist transform when the output is to be sorted?
That keeps new code to a minimum, and the they would need to be
transformed to an intermediate structure to be sorted anyway.

A patch implementing this suggestion is attached. Here are the same
benchmarks with the new patch applied:

With json-object-type set to hash-table:

 (benchmark-run 100 (json-encode huge))
      ⇒ (17.229601604 750 8.015517397999995)

With json-object-type set to plist:

 (benchmark-run 100 (json-encode huge))
      ⇒ (14.363185863 1101 6.992225689000007)

-- Simen

[Message part 2 (text/html, inline)]

[0001-Enable-sorting-of-JSON-object-keys-when-encoding.patch (text/x-patch, attachment)]

Message #14 received at 21616 <at> debbugs.gnu.org (full text, mbox):

From: Dmitry Gutov <dgutov <at> yandex.ru>
To: Simen Heggestøyl <simenheg <at> gmail.com>
Cc: 21616 <at> debbugs.gnu.org
Subject: Re: bug#21616: [PATCH] Enable sorting of JSON object keys when
 encoding
Date: Thu, 12 Nov 2015 04:39:02 +0200

Hi Simen,

On 11/11/2015 08:59 PM, Simen Heggestøyl wrote:

> How about keeping the old encoding code as the default, and only do the
> {hash-table, plist} → alist transform when the output is to be sorted?
> That keeps new code to a minimum, and the they would need to be
> transformed to an intermediate structure to be sorted anyway.
>
> A patch implementing this suggestion is attached. Here are the same
> benchmarks with the new patch applied:

LGTM, please install. Thanks.

Message #19 received at 21616-done <at> debbugs.gnu.org (full text, mbox):

From: Simen Heggestøyl <simenheg <at> gmail.com>
To: Dmitry Gutov <dgutov <at> yandex.ru>
Cc: 21616-done <at> debbugs.gnu.org
Subject: Re: bug#21616: [PATCH] Enable sorting of JSON object keys when
 encoding
Date: Thu, 12 Nov 2015 18:37:33 +0100

[Message part 1 (text/plain, inline)]

Thanks for your time, Dmitry. Installed!

On Thu, Nov 12, 2015 at 3:39 AM, Dmitry Gutov <dgutov <at> yandex.ru> wrote:
> Hi Simen,
> 
> On 11/11/2015 08:59 PM, Simen Heggestøyl wrote:
> 
>> How about keeping the old encoding code as the default, and only do 
>> the
>> {hash-table, plist} → alist transform when the output is to be 
>> sorted?
>> That keeps new code to a minimum, and the they would need to be
>> transformed to an intermediate structure to be sorted anyway.
>> 
>> A patch implementing this suggestion is attached. Here are the same
>> benchmarks with the new patch applied:
> 
> LGTM, please install. Thanks.

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.