bug#67536: 29.1; Calc mode's math-read-preprocess-string conses unnecessarily

[Raffael Stocker]

Org table re-calculation is very slow, partly due to
math-read-preprocess-string of calc mode consing unnecessarily.  For
example, in one (large) table, I get the following memory usage from the
profiler:

...
     60,252,646  96%   - org-ctrl-c-ctrl-c
     60,248,166  96%    - org-table-calc-current-TBLFM
     60,216,431  96%     - funcall-interactively
     60,205,119  96%      - org-table-recalculate
     49,094,651  78%       - org-table-eval-formula
     32,624,631  52%        - calc-eval
     32,624,631  52%         - calc-do-calc-eval
     32,620,487  52%          - calc-do-calc-eval
     32,611,151  52%           - math-read-exprs
     29,388,838  47%            + math-read-preprocess-string
      2,343,257   3%            + math-read-expr-list
...

The reason for the slow-down seems to be that
math-read-preprocess-string conses a lot, keeping the GC busy.  This is
due to heavy use of replace-regexp-in-string in this function:

(defun math-read-preprocess-string (str)
  "Replace some substrings of STR by Calc equivalents."
  (setq str
        (replace-regexp-in-string (concat "[" math-read-superscripts "]+")
                                  "^(\\&)" str))
  (setq str
        (replace-regexp-in-string (concat "[" math-read-subscripts "]+")
                                  "_(\\&)" str))
  (let ((rep-list math-read-replacement-list))
    (while rep-list
      ;; consing like a mad-man here:
      (setq str
            (replace-regexp-in-string (nth 0 (car rep-list))
                                      (nth 1 (car rep-list)) str))
      (setq rep-list (cdr rep-list))))
  str)

I would like to propose using a temp buffer instead of kneading the
string into submission:

(defun math-read-preprocess-string (str)
  "Replace some substrings of STR by Calc equivalents."
  (with-temp-buffer
    (insert str)
    (goto-char 0)
    (while (re-search-forward (concat "[" math-read-superscripts "]+") nil t)
      (replace-match "^(\\&)"))
    (goto-char 0)
    (while (re-search-forward (concat "[" math-read-subscripts "]+") nil t)
      (replace-match "_(\\&)"))
    (goto-char 0)
    (let ((rep-list math-read-replacement-list))
      (while rep-list
        (while (re-search-forward (nth 0 (car rep-list)) nil t)
          (replace-match (nth 1 (car rep-list))))
        (goto-char 0)
        (setq rep-list (cdr rep-list))))
    (buffer-string)))

With this replacement, the profiler shows much less memory usage on the
same org table:

...
     30,411,804  91%   - org-ctrl-c-ctrl-c
     30,407,324  91%    - org-table-calc-current-TBLFM
     30,363,932  91%     - funcall-interactively
     30,331,900  91%      - org-table-recalculate
     20,430,223  61%       - org-table-eval-formula
      6,751,852  20%        + org-table-justify-field-maybe
      4,598,523  13%        - calc-eval
      4,586,091  13%         - calc-do-calc-eval
      4,569,619  13%          - calc-do-calc-eval
      4,547,971  13%           - math-read-exprs
      2,297,453   6%            + math-read-expr-list
      1,347,377   4%            + math-read-preprocess-string
          7,296   0%              math-read-token
...

Only I am not sure the replacement is correct in all possible edge cases
(or whether it uses so much less memory because I have overlooked
something).

What do you think? Can a calc-mode expert weigh in here?

Regards,
Raffael