Re: Parallel job instance identifiers?

[Howard Chu]

Paul Smith wrote:
> On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and
> discussion for GNU make wrote:
>> I'm looking for a way to expose a job ID number to the individual
>> jobs, in a consecutive range from 1-[number of jobs]. Not just
>> unique numbers, for that we could just use $$ already. The purpose
>> is to e.g. assign non-overlapping network port numbers when firing
>> off a number of client/server test scripts in parallel.
>>
>> Is there anything that could do this now, exposed as a Make variable?
> 
> I don't see any way make can provide something like this, at least not
> using the model it does today.  The current requirement is that each
> read from the pipe is of exactly 1 byte and there's no way to guarantee
> a unique value in a single byte of data if there are >256 possible
> values of course.
> 
> One possibility would be for make to write out bytes with values
> 0,1,2,3... then when it gets to 255 start over again: this would give
> unique values for any -j <= 256 and you'd only have to worry about
> wrapping for very large values of -j.

We could change the tokens to be 2 bytes. Individual read/write syscalls
are still guaranteed to be atomic, so there's no danger of interleaving.
The only problem then is ensuring a large enough pipe buffer to avoid
premature blocking.

> The other downside of using a semaphore is I was thinking of
> implementing a "fast fail" feature where as soon as any make in the
> jobserver domain died all the other makes would stop quickly as well.
>  This is something people seem to really want.  One way to implement
> this is to write "error bytes" into the jobserver pipe when we've hit
> "failure mode"; whenever a make instance reads the "error byte" instead
> of the "normal byte" it would finish any outstanding jobs then stop.

It'd be easier just to close the write-side of the pipe, and then all
subsequent read attempts would fail.

> This is pretty straightforward although it's not ideal: make can
> potentially do a lot of work with its single token without trying to
> read from the jobserver for example.
> 
> Of course a single semaphore can't help here.  If we go with "named X"
> we'll need a directory to hold it; one idea would be to write a
> "failed" file into that directory that all make's could check for.

Yes, this is another advantage of using pipes, no special considerations
about namespaces, filesystem or otherwise.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/