[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Cooperation with distributed job processing systems
From: |
SF Markus Elfring |
Subject: |
Re: Cooperation with distributed job processing systems |
Date: |
Mon, 05 Jan 2015 18:24:11 +0100 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 |
> Jobserver is a method of communicating how many jobs make thinks are
> running between different instances (parent/child) of the make
> program itself, so it knows that no more than N jobs are invoked
> between all instances.
How do you think about to delegate such an implementation detail
to a configurable job submission system?
> What you seem to be suggesting is something different:
> a facility for make to start jobs on remote systems.
Yes. - My suggestion goes into this direction.
There are programming interfaces available which support transparent
data exchange between local and remote information systems.
> There are various methods for this: the most common is to use distcc in
> conjunction with make: https://code.google.com/p/distcc/ although this
> is really restricted to building C/C++ programs
It is nice when remote compilation can be used with dedicated tools already.
> so not generally applicable.
I would appreciate when this limitation can be dropped eventually.
> There is a nascent facility in the GNU make code that provides for make
> invoking jobs through a separate facility: the "remote" capability.
I am curious to know a bit more about such a possibility.
> Currently there is a remote-stub.c which defines the interface, and a
> remote-cstms.c which attempts to use the Customs facility for starting
> remote jobs. However, this is obsolete and I'm not sure if anyone has
> tried to use it for years.
Interesting background information ...
> Nevertheless if one were to try to add native facilities for starting
> jobs on remote systems into GNU make itself, rather than doing it
> through use of a special SHELL variable setting, etc. then remote would
> likely be the best way to do it, not jobserver.
Does the software use a clean internal API already?
> I urge you to explore these other methods first, though, as they're a
> lot less effort all the way around. If you can write a small program
> that will forward a command to a remote system, then you can just
> replace SHELL in your makefile with that command:
>
> SHELL = /bin/submit-job
>
> all:
> run-a-job
I find that this approach would apply one step too late because the make
program manages corresponding process parallelisation by the
parameter "-j".
Is an alternative job submission needed earlier?
Regards,
Markus