bug#50733: 28.0.1; project-find-regexp can block Emacs for a long time

[Daniel Martín]

Dmitry Gutov <dgutov@yandex.ru> writes:
>
> IIRC you are using macOS. I received another report recently that
> find/grep based tooling, and project-find-regexp in particular, are 
> pretty slow on that OS.

Yes, this is on macOS.

>
> When you say "block for a long time", how long are we talking about?
>>
> To try it, evaluate
>
>   (benchmark 1 '(project-find-regexp "new-collection"))

I usually work on a monorepo with ~67000 tracked files (many of them big
binary files).  Here's what I get when using ripgrep as the xref search
program:

Elapsed time: 36.087181s (8.067474s in 22 GCs)

Running the same search with ripgrep from the command line takes around
6 seconds.

>
> Another benchmark to try is
>
>   (benchmark 1 '(project-files (project-current)))

Elapsed time: 1.590223s (0.432372s in 1 GCs)

Here's an ELisp profile of the first benchmark:

        8696  78% - command-execute
        8696  78%  - call-interactively
        8493  76%   - funcall-interactively
        8480  76%    - eval-expression
        8479  76%     - eval
        8479  76%      - project-find-regexp
        8227  74%       - xref--show-xrefs
        8227  74%        - xref--show-xref-buffer
        5584  50%         - #<compiled 0x140b5a40100bafc6>
        5584  50%          - apply
        5584  50%           - project--find-regexp-in-files
        5574  50%            - xref-matches-in-files
        3016  27%             - xref--convert-hits
        3000  27%              - mapcan
        2992  27%               - #<compiled -0x6cdcd56218925c3>
        2734  24%                - xref--collect-matches
        2094  18%                 - xref--collect-matches-1
         800   7%                  + xref-make-match
         774   7%                  + xref-make-file-location
         104   0%                   xref--find-file-buffer
          80   0%                   file-remote-p
          51   0%                   xref--regexp-syntax-dependent-p
         906   8%             + xref--process-file-region
         331   2%               sort
        1413  12%         + xref--analyze
        1230  11%         + xref--show-common-initialize
         249   2%       + project-files
           3   0%       + project-current
           9   0%    + minibuffer-complete
           4   0%    + execute-extended-command
         203   1%   + byte-code
        2314  20% - ...
        2314  20%    Automatic GC
          27   0% + timer-event-handler

The search time is reduced when I use a more specific search term,
presumably because the number of results is lower and the Elisp
post-processing takes less time.  Here's what I got, for example, when I
search for something with results from only one file:

Elapsed time: 6.859815s (0.864738s in 2 GCs)

Compared to the time taken by the same query from the command line
(6.5s) shows that the Elisp post-processing time is probably negligible
in this scenario.