Re: Make and Clusters

[Paul Smith]

On Tue, 2017-10-10 at 15:53 +0200, David Delahaye wrote:
> I wonder how the make command is compatible with clusters. I know the
> "-j" option, which allows us to run several jobs simultaneously and
> uses the several cores of a given processor. But if we have several
> nodes of computation of a cluster, is the command "make -j" able to
> run over these nodes (transparently for the user)? I know there are
> specific commands to run jobs on a cluster, I would like to know how
> "make -j" is compatible with these commands.

GNU make doesn't have any built-in support for building on clusters. 
To do so it would have to accept all sorts of other arguments which
defined what machines were in the cluster, what authentication methods
are needed to access the cluster, etc. for all different sorts of
cluster management software, including plain ssh.  Clearly (from the
GNU make manual :)) it doesn't provide any of that.

However, if you have cluster management software you can just change
the commands invoked by make to run those commands in a cluster
instead.  There are many ways to do it: for one, you can write a
wrapper script around your compiler and set the make variable to run
that wrapper instead.  Your wrapper script will send the compilation
job to the cluster, and stay alive until the job has been completed so
it can tell make it's done and what the result was (success/failure).

You can also use other facilities such as distcc http://distcc.org/ to
distribute compilation: these are often designed to work well with GNU
make's -j option.