Subject: bug#67536: closed (Re: bug#67536: 29.1; Calc mode's math-read-preprocess-string conses unnecessarily)

From: [email-masked] (GNU bug Tracking System)
Date: Tue, 19 Dec 2023 18:16:02 +0000

Your bug report

#67536: 29.1; Calc mode's math-read-preprocess-string conses unnecessarily

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
67536: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67536
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#67536: 29.1; Calc mode's math-read-preprocess-string conses unnecessarily
From: Mattias Engdegård <[email-masked]>
To: Raffael Stocker <[email-masked]>
Cc: Eli Zaretskii <[email-masked]>, [email-masked]
Date: Tue, 19 Dec 2023 19:15:38 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

19 dec. 2023 kl. 18.09 skrev Raffael Stocker <[email-masked]>:

> Yes, I think this will be fine.  Thanks!

Closing.

Subject: 29.1; Calc mode's math-read-preprocess-string conses unnecessarily
From: Raffael Stocker <[email-masked]>
To: [email-masked]
Date: Wed, 29 Nov 2023 22:29:38 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

Org table re-calculation is very slow, partly due to
math-read-preprocess-string of calc mode consing unnecessarily.  For
example, in one (large) table, I get the following memory usage from the
profiler:

...
     60,252,646  96%   - org-ctrl-c-ctrl-c
     60,248,166  96%    - org-table-calc-current-TBLFM
     60,216,431  96%     - funcall-interactively
     60,205,119  96%      - org-table-recalculate
     49,094,651  78%       - org-table-eval-formula
     32,624,631  52%        - calc-eval
     32,624,631  52%         - calc-do-calc-eval
     32,620,487  52%          - calc-do-calc-eval
     32,611,151  52%           - math-read-exprs
     29,388,838  47%            + math-read-preprocess-string
      2,343,257   3%            + math-read-expr-list
...

The reason for the slow-down seems to be that
math-read-preprocess-string conses a lot, keeping the GC busy.  This is
due to heavy use of replace-regexp-in-string in this function:

(defun math-read-preprocess-string (str)
  "Replace some substrings of STR by Calc equivalents."
  (setq str
        (replace-regexp-in-string (concat "[" math-read-superscripts "]+")
                                  "^(\\&)" str))
  (setq str
        (replace-regexp-in-string (concat "[" math-read-subscripts "]+")
                                  "_(\\&)" str))
  (let ((rep-list math-read-replacement-list))
    (while rep-list
      ;; consing like a mad-man here:
      (setq str
            (replace-regexp-in-string (nth 0 (car rep-list))
                                      (nth 1 (car rep-list)) str))
      (setq rep-list (cdr rep-list))))
  str)

I would like to propose using a temp buffer instead of kneading the
string into submission:

(defun math-read-preprocess-string (str)
  "Replace some substrings of STR by Calc equivalents."
  (with-temp-buffer
    (insert str)
    (goto-char 0)
    (while (re-search-forward (concat "[" math-read-superscripts "]+") nil t)
      (replace-match "^(\\&)"))
    (goto-char 0)
    (while (re-search-forward (concat "[" math-read-subscripts "]+") nil t)
      (replace-match "_(\\&)"))
    (goto-char 0)
    (let ((rep-list math-read-replacement-list))
      (while rep-list
        (while (re-search-forward (nth 0 (car rep-list)) nil t)
          (replace-match (nth 1 (car rep-list))))
        (goto-char 0)
        (setq rep-list (cdr rep-list))))
    (buffer-string)))

With this replacement, the profiler shows much less memory usage on the
same org table:

...
     30,411,804  91%   - org-ctrl-c-ctrl-c
     30,407,324  91%    - org-table-calc-current-TBLFM
     30,363,932  91%     - funcall-interactively
     30,331,900  91%      - org-table-recalculate
     20,430,223  61%       - org-table-eval-formula
      6,751,852  20%        + org-table-justify-field-maybe
      4,598,523  13%        - calc-eval
      4,586,091  13%         - calc-do-calc-eval
      4,569,619  13%          - calc-do-calc-eval
      4,547,971  13%           - math-read-exprs
      2,297,453   6%            + math-read-expr-list
      1,347,377   4%            + math-read-preprocess-string
          7,296   0%              math-read-token
...

Only I am not sure the replacement is correct in all possible edge cases
(or whether it uses so much less memory because I have overlooked
something).

What do you think? Can a calc-mode expert weigh in here?

Regards,
Raffael