Re: Cooperation with distributed job processing systems

[Paul Smith]

On Mon, 2015-01-05 at 18:24 +0100, SF Markus Elfring wrote:
> > Jobserver is a method of communicating how many jobs make thinks are
> > running between different instances (parent/child) of the make
> > program itself, so it knows that no more than N jobs are invoked
> > between all instances.
> 
> How do you think about to delegate such an implementation detail
> to a configurable job submission system?

That could be done, but the current implementation of the jobserver
software isn't really separable like that.  It would be a nice thing to
make a more abstract API which could be replaced with alternatives.

> > Nevertheless if one were to try to add native facilities for starting
> > jobs on remote systems into GNU make itself, rather than doing it
> > through use of a special SHELL variable setting, etc. then remote would
> > likely be the best way to do it, not jobserver.
> 
> Does the software use a clean internal API already?

As I said, the API is defined in remote-stub.c.  It's relatively clean.
I don't think anyone has used it for a very long time so it's quite
possible it's bit-rotted.

> > I urge you to explore these other methods first, though, as they're a
> > lot less effort all the way around.  If you can write a small program
> > that will forward a command to a remote system, then you can just
> > replace SHELL in your makefile with that command:
> > 
> >     SHELL = /bin/submit-job
> > 
> >     all:
> >             run-a-job
> 
> I find that this approach would apply one step too late because the make
> program manages corresponding process parallelisation by the
> parameter "-j".

Correct.  There are two completely separate things: one thing is make
deciding how many jobs can be run in parallel and when more jobs can be
started.  That's controlled by the jobserver and currently there's no
way to replace the built-in jobserver implementation with something
different (such as allowing a remote facility to track this).

The second thing is invoking a single job once make believes that it's
OK to do so.  The "remote" API can make it possible to build make so
that it always uses this method, but I think it's become obsolete
because it's simpler and more flexible to implement a separate program
and reset SHELL, as I describe above, than implement a bunch of new code
inside GNU make itself.