WISHLIST: work queues for SMP machines and background jobs

[Austin Donnelly]

Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: Linux
Compiler: gcc
Compilation CFLAGS:   -DHAVE_VFPRINTF -DHAVE_UNISTD_H -DHAVE_STDLIB_H 
-DHAVE_LIMITS_H        -DHAVE_GETGROUPS -DHAVE_RESOURCE -DHAVE_SYS_PARAM 
-DVOID_SIGHANDLER -DHAVE_GETDTABLESIZE -DHAVE_BCOPY -DHAVE_GETPW_DECLS 
-DHAVE_GETHOSTNAME -DHAVE_WAIT_H         -DHAVE_DUP2 -DHAVE_STRERROR  
-DHAVE_DIRENT_H -DHAVE_STRING_H        -DHAVE_VARARGS_H -DHAVE_STRCHR  
-DHAVE_DEV_FD        -Di386 -DLinux -DSHELL -DHAVE_ALLOCA -DHAVE_ALLOCA_H       
-O2 -m486 -fno-strength-reduce
uname output: Linux hornet.cl.cam.ac.uk 2.2.16-4.cl.ext3 #1 Fri Aug 18 14:18:55 
BST 2000 i686 unknown

Bash Version: 1.14
Patch Level: 7

Description:

This describes a WISHLIST/FEATURE REQUEST I'd like considered for
future versions of bash.

Nowadays, multiprocessor machines are increasingly common.  However,
it is often hard to use the processors effectively.

In particular, some tasks are inherently parallel, such as the
encoding of digital audio files.  Given a number of .wav files, it is
trivial to write:

  for i in *.wav; do
    encode $i &
  done
  wait

However on most machines this will perform poorly as there will
typically be fewer CPUs than wav files.  While this loop exposes the
maximum amount of parallelism available, this is too much for most
machines.

My proposal is for bash to find out how many CPUs are present on the
particular system it is running on, and keep a "work queue" of
available jobs.  Bash can then limit the number of running jobs to at
or below the system CPU limit.  To make this facility useful, a
"waitq" command is needed, which waits for the work queue to be empty
and for all related issued commands to have terminated.

So the example given earlier could be written:

  for i in *.wav; do
    encode $i <&>
  done
  waitq

Where "<&>" is some new syntax introduced which means to enqueue the
preceding command on the work queue.  The difference between this
version and the previous is that at most NUM_CPU copies of the encode
process will be started.

I haven't thought much about how pipelines should be handled, but I
suspect that each process in the pipeline should count as a
outstanding process.  So:
  grep foo file1 | wc >results1 <&>
  grep foo file2 | wc >results2 <&>
  waitq
on a 2-CPU machine would run the pipelines one after the other, but on
a 4-CPU box it would run them in parallel.

On a uniprocessor, the <&> syntax and waitq command have no effect,
since there is never an opportunity to issue multiple commands in
parallel.  The <&> notation merely annotates the script to let bash
know where the potential for parallelism exists.

It is possible that more than one work queue might be useful.  This
could be easily supported by allowing <&NAME> to refer to the NAMEd work
queue etc, and "waitq NAME" to wait for that queue to drain.  The
empty string would represent the "default" work queue.

While the motivating example given is fairly simple, the ability to
have such minimal SMP-awareness in bash is probably a useful general
facility.

At the very least, please consider adding an environment variable
(_NUM_CPUS or somesuch) giving portable access to the number of CPUs
available on the current system.

Austin