Re: optimize tramp for for many start-file-process'es in parallel

[Michael Albinus]

Hendrik Tews <hendrik@askra.de> writes:

> Hi,

Hi Hendrik,

> I am maintaining an Emacs library (coq-par-compile in Proof
> General, details below) that manages a number of parallel running
> background jobs, started via `start-file-process'. To decide,
> which jobs need to be started, the library also uses
> `file-attributes', `file-truename' and maybe other
> remote-file-aware functions. When I use the library with a remote
> file, even over /ssh:localhost:, it gets totally slow. For local
> files Emacs stays responsive, while this library is running. For
> remote files Emacs is more or less locked up.
>
> I am not a Tramp expert and I haven't spend much time
> investigating, but from what I found, the problem might be that
> Tramp opens a new ssh channel (using ControlMaster, but still
> new) for each asynchronous (`start-file-process') connection.

Yes. Otherwise, it wouldn't be possible to run the processes asynchronously.

> I also see several repeating blocks of
>
>     00:03:14.329296 tramp-process-one-action (5) # Looking for regexp 
> "\(\(Are you sure you want to continue connecting 
> (yes/no\(?:\(?:/\[fingerprint]\)?)\?\)\)\s-*\)\'" from remote shell

[...]

> which seem to be responsible for the delay of 0.03 sec between
>
>     00:07:26.744746 tramp-process-actions (3) # Waiting for prompts from 
> remote shell...
> and
>
>     00:07:26.774526 tramp-process-actions (3) # Waiting for prompts from 
> remote shell...done

Likely, yes. This is the handshake for every new remote process, looking
for a password, for other questions during login, and alike.

> I would be very grateful for any hints about how to increase the
> performance of my library for remote files over Tramp.

With Tramp 2.5, so called "direct asynchronous processes" have been
introduced. Instead of calling "ssh -l user -o ControlMasterOptions
host" and then looking, whether there is some interactive ping-pong with
the new remote shell, Tramp calls directly "ssh -l user -o
ControlMasterOptions host <command>". This is much faster. There are
some drawbacks, most moteable a missing possibility to provide a
password. Quoting the Tramp manual:

--8<---------------cut here---------------start------------->8---
   Furthermore, this approach has the following limitations:

   • It works only for connection methods defined in ‘tramp-sh.el’ and
     ‘tramp-adb.el’.

   • It does not support interactive user authentication.  With
     ‘ssh’-based methods, this can be avoided by using a password
     agent like ‘ssh-agent’, using public key authentication, or using
     ‘ControlMaster’ options.

   • It cannot be applied for ‘ssh’-based methods, which use the
     ‘RemoteCommand’ option.

   • It cannot be killed via ‘interrupt-process’.

   • It does not report the remote terminal name via
     ‘process-tty-name’.

   • It does not set process property ‘remote-pid’.

   • It does not use ‘tramp-remote-path’.
--8<---------------cut here---------------end--------------->8---

Anyway, you might try it. Install recent Tramp 2.5.0.3 from GNU
ELPA. And activate it via

--8<---------------cut here---------------start------------->8---
     (add-to-list 'tramp-connection-properties
                  (list (regexp-quote "/ssh:user@host:")
                        "direct-async-process" t))
--8<---------------cut here---------------end--------------->8---

> I would even consider using a lower level interface for accessing the
> remote system, where, for instance, I tell Tramp which of the 5 open
> ssh connection it should use for the next `file-attributes' call.

There's nothing to configure. `sile-attributes' is a synchronous
operation, it is performed always on the main Tramp process for a given
host. You could only try to keep Tramp caches, setting
remote-file-name-inhibit-cache to nil might help.

> Thanks in advance,
>
> Hendrik

Best regards, Michael.