bug#66117: 30.0.50; `find-buffer-visiting' is slow when opening large number of buffers

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> > My advice is to have a prototype working, then time it on local
>> > filesystems.
>> 
>> See the attached patch.
>
> Thanks.  This still has some unnecessary overhead (I actually thought
> about moving the entire find-buffer-visiting into C, not replacing it
> with Lisp that calls new primitives).  But this already makes
> find-buffer-visiting take an insignificant portion of CPU time, so I
> think it proves that the idea is workable:

I looked into rewriting `find-buffer-visiting' fully in C, but it
appears to be rather tricky because it calls `file-truename' and
`abbreviate-file-name', which are Elisp functions recursively calling
other Elisp functions.

And `abbreviate-file-name' is the slowest function showing up in the
profiler (with my patch applied):

;; Increased the number of files to get more than ~2 sec CPU samples.
(dotimes (i 3000) (with-temp-file (format "/tmp/test/%d.txt" i) (insert "* This 
is test")))
(dolist (file (directory-files "/tmp/test/" t "txt"))
  (find-file-noselect file))

(cpu-profile-w-patch-3000-files, reverse call tree)

        3130  29%   Automatic GC
        1305  12% + abbreviate-file-name
         845   7% + inhibit-local-variables-p
         842   7% + uniquify-rationalize-file-buffer-names
         719   6% + find-buffer-visiting
         619   5% + locate-dominating-file
         350   3% + dir-locals--all-files
         332   3% + set-auto-mode--apply-alist
         283   2% + hack-local-variables--find-variables
         272   2% + file-truename

I am not sure if we want to go deeper into the rabbit hole, but if we
do, here is also the perf profile:

    25.35%  emacs         emacs                            [.] 
set_default_internal
    15.73%  emacs         emacs                            [.] 
process_mark_stack
    10.18%  emacs         emacs                            [.] 
re_match_2_internal
     8.88%  emacs         emacs                            [.] assq_no_quit
     2.23%  emacs         emacs                            [.] Fstring_equal
     2.01%  emacs         emacs                            [.] 
pdumper_marked_p_impl

The main contributor is set_default_internal. I _suspect_ (because we
have non-linear scaling with the number of buffers - 3x increase in the
number of files lead to ~6x increase in the run time) that the main
contributor here is let-binding `case-fold-search' - special per-buffer
variable with no default value. In particular, `specbind' has the
following comment:

            /* If SYMBOL is a per-buffer variable which doesn't have a
               buffer-local value here, make the `let' change the global
               value by changing the value of SYMBOL in all buffers not
               having their own value.  This is consistent with what
               happens with other buffer-local variables.  */

which implies that the binding will iterate over all the buffers,
leading to non-linear scaling in our scenario.

cpu-profile-w-patch-3000-files
Description: Binary data

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>