Re: feature request: "wait --free-slot" for poor man's parallelization

[Mike Coleman]

On Fri, Oct 3, 2008 at 4:13 AM, Jan Schampera <jan.schampera@web.de> wrote:
> Mike Coleman wrote:
>>
>> Here's a bash feature I'd love to see, but don't have time to
>> implement myself: a "--free-slot" flag to 'wait' that will wait until
>> there is at least one free "slot" available, where a slot is basically
>> a CPU core.
>>
>> Example usage:
>>
>> $ for ((n=0; n<100; n++)); do
>>      my_experiment $n > $n.out &
>>      wait --free-slot 4
>>  done
>>
>> which I would run on a quad-CPU host.  The meaning of the wait is: if
>> I already have four background invocations of "my_experiment" running,
>> wait until one finishes, then proceed.  (If there are fewer than four
>> running, proceed immediately.)
>>
>> This provides a nice sloppy way of letting me parallelize in a script
>> without having to complicate things.  "make -j" and "xargs -P" provide
>> some of this capability, but are a lot more work to set up--I'd really
>> like to have this at my fingertips for interactive stuff.
>
> I did something like that as a script, using an array of PIDs and a big loop
> and so on. I don't think it needs to be part of internal Bash features, it
> wasn't that much work.

I'm sure this could be done with a script, but I'm imagining a
scenario like the above where I don't need to get out an editor or
anything--it'd be a quick-and-dirty thing I could type in at the
command line.  If there's a way to write "wait-free-slot" as a
function (that could be source'd in), I'm all ears, but I don't see
it.  I guess it could poll the current children with ps, but that
seems kind of ugly.

Mike