Re: Examples of concurrent coproc usage?

[Carl Edquist]

On 4/4/24 7:23 PM, Martin D Kealey wrote:

> I'm somewhat uneasy about having coprocs inaccessible to each other. I 
> can foresee reasonable cases where I'd want a coproc to utilize one or 
> more other coprocs.
>
> In particular, I can see cases where a coproc is written to by one 
> process, and read from by another.
>
> Can we at least have the auto-close behaviour be made optional, so that 
> it can be turned off when we want to do something more sophisticated?

With support for multiple coprocs, auto-closing the fds to other coprocs 
when creating new ones is important in order to avoid deadlocks.

But if you're willing to take on management of those coproc fds yourself, 
you can expose them to new coprocs by making your own copies with exec 
redirections.

But this only "kind of" works, because for some reason bash seems to close 
all pipe fds for external commands in coprocs, even the ones that the user 
explicitly copies with exec redirections.

(More on that in a bit.)


On Mon, 8 Apr 2024, Chet Ramey wrote:

> On 4/4/24 7:23 PM, Martin D Kealey wrote:
> > I'm somewhat uneasy about having coprocs inaccessible to each other. I 
> > can foresee reasonable cases where I'd want a coproc to utilize one or 
> > more other coprocs.
>
> That's not the intended purpose,

Just a bit of levity here - i can picture Doc from back to the future 
exclaiming, "Marty, it's perfect!  You're just not thinking 4th 
dimensionally!"

> so I don't think not fixing a bug to accommodate some future 
> hypothetical use case is a good idea. That's why there's a warning 
> message when you try to use more than one coproc -- the shell doesn't 
> keep track of more than one.
>
> If you want two processes to communicate (really three), you might want
> to build with the multiple coproc support and use the shell as the
> arbiter.


For what it's worth, my experience is that coprocesses in bash (rigged up 
by means other than the coproc keyword) become very fun and interesting 
when you allow for the possibility of communication between coprocesses. 
(Most of my use cases for coprocesses fall under this category, actually.)

The most basic commands for tying multiple coprocesses together are tee(1) 
and paste(1), for writing to or reading from multiple coprocesses at once.

You can do this already with process substitutions like

        tee >(cmd1) >(cmd2)

        paste <(cmd3) <(cmd4)


My claim here is that there are uses for this where these commands are all 
separate coprocesses; that is, you'd want to read the output from cmd1 and 
cmd2 separately, and provide input for cmd3 and cmd4 separately.

(I'll try to send some examples in a later email.)


Nevertheless it's still crucial to keep the shell's existing coprocess fds 
out of new coprocesses, otherwise you easily run yourself into deadlock.


Now, if you built bash with multiple coproc support, I would have expected 
you could still rig this up, by doing the redirection work explicitly 
yourself.  Something like this:

        coproc UP   { stdbuf -oL tr a-z A-Z; }
        coproc DOWN { stdbuf -oL tr A-Z a-z; }

        # make user-managed backup copies of coproc fds
        exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
        exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}

        coproc THREEWAY { tee /dev/fd/$up_w  /dev/fd/$down_w; }


But the above doesn't actually work, as it seems that the coproc shell 
(THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even the 
user-managed ones explicitly copied with exec.

As a result, you get back errors like this:

        tee: /dev/fd/11: No such file or directory
        tee: /dev/fd/13: No such file or directory


That's the case even if you do something more explicit like:

        coproc UP_AND_OUT { tee /dev/fd/99  99>&$up_w; }

the '99>&$up_w' redirection succeeds, showing that the coproc does have 
access to its backup fd $up_w (*), but apparently the shell closes fd 99 
(as well as $up_w) before exec'ing the tee command.

Note the coproc shell only does this with pipes; it leaves other user 
managed fds like files or directories alone.

I have no idea why that's the case, and i wonder whether it's intentional 
or an oversight.


But anyway, i imagine that if one wants to use multi coproc support (which 
requires automatically closing the shell's coproc fds for new coprocs), 
and wants to set up multiple coprocs to communicate amongst themselves, 
then the way to go would be explicit redirections.

(But again, this requires fixing this peculiar behavior where the coproc 
shell closes even the user managed copies of pipe fds before exec'ing 
external commands.)


(*) to prove that the coproc shell does have access to $up_w, we can make 
a shell-only replacement for tee(1) :  (actually works)

        fdtee () {
          local line fd
          while read -r line; do
            for fd; do
              printf '%s\n' "$line" >&$fd;
            done;
          done;
        }

        coproc UP   { stdbuf -oL tr a-z A-Z; }
        coproc DOWN { stdbuf -oL tr A-Z a-z; }

        # make user-managed backup copies of coproc fds
        exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
        exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}

        stdout=1
        coproc THREEWAY { fdtee $stdout $up_w $down_w; }

        # save these too, for safe keeping
        exec {tee_r}<&${THREEWAY[0]} {tee_w}>&${THREEWAY[1]}


Then:  (actually works)

        $ echo 'Greetings!' >&$tee_w
        $ read -u $tee_r  plain
        $ read -u $up_r   upped
        $ read -u $down_r downed
        $ echo "[$plain] [$upped] [$downed]"
        [Greetings!] [GREETINGS!] [greetings!]


This is a pretty trivial example just to demonstrate the concept.  But 
once you have the freedom to play with it, you find more interesting, 
useful applications.

Of course, for the above technique to be generally useful, external 
commands need access to these user-managed fds (copied with exec).  (I 
have no idea why the coproc shell closes them.)  The shell is crippled 
when limited to builtins.

(I'll try to tidy up some working examples with my coprocess management 
library this week, for the curious.)


Juicy thread hey?  I can hardly keep up!  :)

Carl