[O] Bug: Org Table Performance Issue/Regression

[Tim Baumgard]

I recently updated Emacs from 25.3 (Org mode v. 8.2.10) to 26.1 (Org mode
v. 9.1.9) and noticed a performance issue/regression in one of my Org files that
contains a table that currently has nine columns, around 1,200 rows, and uses
formulas. In 25.3, I could create a new row or use org-table-copy-down without
issues. In 26.1, there’s a multi-second delay each time I try to do either of
these things. As mentioned below, this issue also occurs in the latest commits
to both Emacs and Org mode.

I did some profiling and found that garbage-collect was being called quite a
bit. I set gc-cons-threshold in both versions to a large value to effectively
disable garbage collection and noticed two things: 1) it largely fixed the
performance issues—though performance still wasn’t as good as it was in 25.3—and
2) Emacs/Org mode was using about four times as much memory as it did after
creating a new row and doing calling org-table-copy-down three
times. Specifically, Emacs 25.3 was using a little under 200 MB and Emacs 26.3
was using a around 800 MB.

I also noticed that Org mode is recalculating the table formulas more often than
in past versions. It used to be that it would only recalculate the formulas for
the whole table when I called org-table-recalculate, but after testing some
things and then diff’ing the files, I noticed that other rows were being updated
based on the table formulas I’m using. I’m unsure if this is related.

For what it’s worth, using the Org mode files from Emacs 25.3 in 26.1 doesn’t
display the performance issue, but there are other issues that make it a poor
workaround. What I’m currently doing to work around this for my use case is,
only for this file, setting gc-cons-threshold to a large value and adding a hook
to after-save-hook to call garbage-collect since calling it once when saving
doesn’t take too long.

I’ve inlined a test file and a test init.el file below. The test file includes a
reduced version of the table I’m currently using, including the table TBLFM
line. The init.el file includes the functions required for my table formulas and
how I was using gc-cons-threshold and my workaround. I was still seeing the
issue even without using the formulas, but I thought I'd include them in case
it's helpful.

Some notes:
- All of my testing was done using emacs -Q
- I’m running Emacs on macOS compiled from the sources available on gnu.org. 
- I also tested everything using the latest commits of Emacs
  (efa750e68466fb1bab03028701350a745e0504b6) and Org mode
        (842002f9f7e3246cc285cc9bca1adb715120438d) from the repositories.
- Emacs: GNU Emacs 27.0.50 (build 1, x86_64-apple-darwin17.6.0, NS
  appkit-1561.40 Version 10.13.5 (Build 17F77))
        of 2018-06-10
- Package: Org mode version 9.1.13 (release_9.1.13-791-g842002)


test.org
======

|   | A       | B         | C      | D                         | E              
           |        F |   G | H                          |
|---+---------+-----------+--------+---------------------------+---------------------------+----------+-----+----------------------------|
| # | ABCDEFG | HIJKLMNOP | QRSTUV | [2018-06-10 Sun 00:00:00] | [2018-06-10 
Sun 02:00:00] | 02:00:00 | 2.0 | ABCDEFGHIJKLMNOPQRSTUVWXYZ |
#+TBLFM: $7='(time-stamp-difference $5 $6)::$8='(time-stamp-difference-hours $5 
$6)


init.el
====

(defun time-stamp-difference (earlier-date later-date)
  "Get the difference from two time stamps. This only works when the difference 
is less than 24 hours."
        (let* ((time-difference
                                 (round (- (time-to-seconds (date-to-time 
later-date))
                                                                         
(time-to-seconds (date-to-time earlier-date)))))
                                (absolute-time-difference (max time-difference 
0)))
                (format-time-string "%T" (encode-time absolute-time-difference 
0 0 1 1 1))))

(defun time-stamp-difference-hours (earlier-date later-date)
  "Get the difference from two time stamps in the number of hours."
        (let* ((seconds
                                 (- (time-to-seconds (date-to-time later-date))
                                                (time-to-seconds (date-to-time 
earlier-date))))
                                 (absolute-seconds (max seconds 0))
                                 (hours (/ (/ absolute-seconds 60) 60.0)))
                (/ (round (* hours 1000.0)) 1000.0)))

;; This sets the garbage-collection threshold to a large value, effectively
;; disabling it for testing purposes.
;; (setq-default gc-cons-threshold 80000000000)
;; (add-hook 'after-save-hook 'garbage-collect)