Re: feature request: "wait --free-slot" for poor man's parallelization

[Bob Proulx]

Mike Coleman wrote:
> Jan Schampera wrote:
> > Mike Coleman wrote:
> >> Here's a bash feature I'd love to see, but don't have time to
> >> implement myself: a "--free-slot" flag to 'wait' that will wait until
> >> there is at least one free "slot" available, where a slot is basically
> >> a CPU core.

If you start working with compute queues you will find that there are
endless different ways that people want to define job slots.  It isn't
a simple problem.

> >> which I would run on a quad-CPU host.  The meaning of the wait is: if
> >> I already have four background invocations of "my_experiment" running,
> >> wait until one finishes, then proceed.  (If there are fewer than four
> >> running, proceed immediately.)

Of course you can code this yourself.  Bash is a general purpose shell
and programming language.  Should it really contain such domain
specific things in it?  If so then it would open the door to a zillion
and one other domain specific tasks that are equal to this one.  I
think the above is better as a code library similar to the completion
library that can be include as desired.

> > I did something like that as a script, using an array of PIDs and
> > a big loop and so on. I don't think it needs to be part of
> > internal Bash features, it wasn't that much work.
> 
> I'm sure this could be done with a script, but I'm imagining a
> scenario like the above where I don't need to get out an editor or
> anything--it'd be a quick-and-dirty thing I could type in at the
> command line.

Something similar to this?

  source /etc/bash_completion

People have written everything from debuggers to directory stack
libraries for the shell.  A parallel job library seems similar.  It
seems best as a code library.

> If there's a way to write "wait-free-slot" as a function (that could
> be source'd in), I'm all ears, but I don't see it.  I guess it could
> poll the current children with ps, but that seems kind of ugly.

I have written such a thing in Perl and am aware of the issues.  I am
sure something like it could be useful to many people in bash too if
you decided to write it and contribute it.

Bob