[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cooperation with distributed job processing systems

From: SF Markus Elfring
Subject: Re: Cooperation with distributed job processing systems
Date: Mon, 05 Jan 2015 18:24:11 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0

> Jobserver is a method of communicating how many jobs make thinks are
> running between different instances (parent/child) of the make
> program itself, so it knows that no more than N jobs are invoked
> between all instances.

How do you think about to delegate such an implementation detail
to a configurable job submission system?

> What you seem to be suggesting is something different:
> a facility for make to start jobs on remote systems.

Yes. - My suggestion goes into this direction.

There are programming interfaces available which support transparent
data exchange between local and remote information systems.

> There are various methods for this: the most common is to use distcc in
> conjunction with make: https://code.google.com/p/distcc/ although this
> is really restricted to building C/C++ programs

It is nice when remote compilation can be used with dedicated tools already.

> so not generally applicable.

I would appreciate when this limitation can be dropped eventually.

> There is a nascent facility in the GNU make code that provides for make
> invoking jobs through a separate facility: the "remote" capability.

I am curious to know a bit more about such a possibility.

> Currently there is a remote-stub.c which defines the interface, and a
> remote-cstms.c which attempts to use the Customs facility for starting
> remote jobs.  However, this is obsolete and I'm not sure if anyone has
> tried to use it for years.

Interesting background information ...

> Nevertheless if one were to try to add native facilities for starting
> jobs on remote systems into GNU make itself, rather than doing it
> through use of a special SHELL variable setting, etc. then remote would
> likely be the best way to do it, not jobserver.

Does the software use a clean internal API already?

> I urge you to explore these other methods first, though, as they're a
> lot less effort all the way around.  If you can write a small program
> that will forward a command to a remote system, then you can just
> replace SHELL in your makefile with that command:
>     SHELL = /bin/submit-job
>     all:
>             run-a-job

I find that this approach would apply one step too late because the make
program manages corresponding process parallelisation by the
parameter "-j".

Is an alternative job submission needed earlier?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]