bug#49264: 28.0.50; project.el+tramp performance issue

[Ergus]

On Wed, Jun 30, 2021 at 03:46:26PM +0300, Eli Zaretskii wrote:
> > Date: Wed, 30 Jun 2021 00:21:42 +0200
> > From: Ergus <spacibba@aol.com>
> > Cc: 49264@debbugs.gnu.org
> >
> > >AFAICT, most of the time is taken by 'apply', but the profile doesn't
> > >show which function is called by 'apply'.  Can you tell which function
> > >is that?
> > >
> > apply is called in vc-responsible-backend and looking at the percentages
> > it is obvious that the times are going in
> >
> >    23%                      + vc-svn-responsible-p
> >    18%                      + vc-bzr-responsible-p
> >    15%                      + vc-hg-responsible-p
> >     6%                      + vc-git-responsible-p
> >     2%                      + vc-cvs-responsible-p
> >     2%                      + vc-rcs-responsible-p
> >     1%                      + vc-sccs-responsible-p
> >     1%                      + vc-src-responsible-p
> > ------
> >    68%
> >
> > These are the backends' functions that are passed to vc-call-backend in
> > the mapcar in vc-responsible-backend and the output of
> > vc-find-backend-function.
>
> But if you still need to wait for dozens of seconds with just 2
> backends, which take only 20% of the time according to the above, then
> how do you want to speed this up in your case?  How about not using
> ivy (which AFAICT is the root cause of this slowness)?
In the test Dimitry requested I sent the numbers of eval:

(benchmark 1 '(project-current))

Only with git and mercurial it takes:

Elapsed time: 3.018998s (0.109912s in 1 GCs)

With the entire list in vc-handled-backends

Elapsed time: 8.197923s (0.507396s in 6 GCs)

So, ivy is not the problem.

> > >... after removing the "unused" VC back-ends, you say that the code
> > >still runs very slowly.  So is the issue with VC back-ends still
> > >relevant, and if so, how?
> > >
> > Yes, it is. It is still slow, lets say around 20-40 seconds to complete
> > the command. But that's that's much faster than before (3-5 minutes);
> > but still too slow to make the command usable.
>
> It's still unacceptably slow, so that couldn't be a solution,
> certainly not a general one, IMO.
Agree, but it is another symptom about where the problem is.

> > As a note here, when N files are in the same directory the normal thing
> > is that all of them share the VCS. So calling a check function for all
> > of them is redundant and slow.
>
> AFAIR, that's not really true, and ISTR project.el aims to support the
> use cases with several different VC backends.
I can't speak for all the possible configurations; but IMO the most
common configurations around are projects with single vcs OR projects
with subprojects (git submodules) in different subdirs I am not aware of
any project where different vcs are in the same root directory... For
sure I have never seen that...

For example something like:

project0
|- file0-0
|- dir0
|   |- file0-1
|   |- file0-1
|
|- subproject1
|  |- file1-1
|  |- file1-2
|  |- dir1
|
|- subproject2
   |- file2-1
   |- file2-2
   |- dir2
   |  |- file2-3
   |
   |- subproject3
      |- file3-1
      |- dir2
         |- file3-2


Again: IMHO if we support this configuration we support 99% of the
projects around. And I already proposed an option (using the default-dir
as cache index) that already cover this with no functionality penalty.

> And again, waiting for 30 seconds when you have just 10 buffers is
> unacceptable.  E.g., in the session where I'm writing this, I have 80
> buffers that visit files in a single branch of the Emacs Git
> repository, almost an order of magnitude more than your 10 buffers.
With my solution we only impact performance:
1) The first time we call the command
AND
2) All the 80 files are all in different directories.

Otherwise there will be some benefit either for files located in the
same directory and/or the next time you call the command (or open a file
in one of the directories emacs already knows about.).

> So we must find some better way of getting reasonable performance in
> this use case.  If the round-trip of the VC backend to a remote
> filesystem is the bottleneck, let's try to speed that up in some way.

In case you are interested about the issue with the external package:

https://github.com/FelipeLema/emacs-counsel-gtags/issues/29