Re: Parallel job instance identifiers?

[David Boyce]

> One possibility would be for make to write out bytes with values 0,1,2,3... then when it gets to 255 start over again ...

I hope this sin't a dumb question, I've never worked with the job server, but hy not use the value of the file descriptor from the parent side as the token to send down the pipe? Still limited to 255 of course, same wrapping problem, but the fd is guaranteed unique and I could imagine cases where that data might come in handy for debugging via strace or whatever.

David

On Mon, Sep 6, 2021 at 7:37 AM Paul Smith <psmith@gnu.org> wrote:

On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and
discussion for GNU make wrote:
> I'm looking for a way to expose a job ID number to the individual
> jobs, in a consecutive range from 1-[number of jobs]. Not just
> unique numbers, for that we could just use $$ already. The purpose
> is to e.g. assign non-overlapping network port numbers when firing
> off a number of client/server test scripts in parallel.
>
> Is there anything that could do this now, exposed as a Make variable?

I don't see any way make can provide something like this, at least not
using the model it does today.  The current requirement is that each
read from the pipe is of exactly 1 byte and there's no way to guarantee
a unique value in a single byte of data if there are >256 possible
values of course.

One possibility would be for make to write out bytes with values
0,1,2,3... then when it gets to 255 start over again: this would give
unique values for any -j <= 256 and you'd only have to worry about
wrapping for very large values of -j.

One reason I'm not excited about this idea is that I've been seriously
contemplating switching from the current pipe-based jobserver control
method, to using POSIX named semaphores instead (at least as the
default).  There is support for these in most systems these days, and
it's already how Windows works (with the Windows equivalent).  The
advantage here is it's just easier in lots of ways to not have to pass
open file descriptors between make invocations.  I could use a named
pipe but semaphores are more obviously built for this.  But of course
there's no possibility of uniquely identifying a semaphore lock.

The other downside of using a semaphore is I was thinking of
implementing a "fast fail" feature where as soon as any make in the
jobserver domain died all the other makes would stop quickly as well.
 This is something people seem to really want.  One way to implement
this is to write "error bytes" into the jobserver pipe when we've hit
"failure mode"; whenever a make instance reads the "error byte" instead
of the "normal byte" it would finish any outstanding jobs then stop.

This is pretty straightforward although it's not ideal: make can
potentially do a lot of work with its single token without trying to
read from the jobserver for example.

Of course a single semaphore can't help here.  If we go with "named X"
we'll need a directory to hold it; one idea would be to write a
"failed" file into that directory that all make's could check for.