bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Examples of concurrent coproc usage?


From: Carl Edquist
Subject: Re: Examples of concurrent coproc usage?
Date: Tue, 9 Apr 2024 10:58:41 -0500 (CDT)

On 4/4/24 7:23 PM, Martin D Kealey wrote:

I'm somewhat uneasy about having coprocs inaccessible to each other. I can foresee reasonable cases where I'd want a coproc to utilize one or more other coprocs.

In particular, I can see cases where a coproc is written to by one process, and read from by another.

Can we at least have the auto-close behaviour be made optional, so that it can be turned off when we want to do something more sophisticated?

With support for multiple coprocs, auto-closing the fds to other coprocs when creating new ones is important in order to avoid deadlocks.

But if you're willing to take on management of those coproc fds yourself, you can expose them to new coprocs by making your own copies with exec redirections.

But this only "kind of" works, because for some reason bash seems to close all pipe fds for external commands in coprocs, even the ones that the user explicitly copies with exec redirections.

(More on that in a bit.)


On Mon, 8 Apr 2024, Chet Ramey wrote:

On 4/4/24 7:23 PM, Martin D Kealey wrote:
I'm somewhat uneasy about having coprocs inaccessible to each other. I can foresee reasonable cases where I'd want a coproc to utilize one or more other coprocs.

That's not the intended purpose,

Just a bit of levity here - i can picture Doc from back to the future exclaiming, "Marty, it's perfect! You're just not thinking 4th dimensionally!"

so I don't think not fixing a bug to accommodate some future hypothetical use case is a good idea. That's why there's a warning message when you try to use more than one coproc -- the shell doesn't keep track of more than one.

If you want two processes to communicate (really three), you might want
to build with the multiple coproc support and use the shell as the
arbiter.


For what it's worth, my experience is that coprocesses in bash (rigged up by means other than the coproc keyword) become very fun and interesting when you allow for the possibility of communication between coprocesses. (Most of my use cases for coprocesses fall under this category, actually.)

The most basic commands for tying multiple coprocesses together are tee(1) and paste(1), for writing to or reading from multiple coprocesses at once.

You can do this already with process substitutions like

        tee >(cmd1) >(cmd2)

        paste <(cmd3) <(cmd4)


My claim here is that there are uses for this where these commands are all separate coprocesses; that is, you'd want to read the output from cmd1 and cmd2 separately, and provide input for cmd3 and cmd4 separately.

(I'll try to send some examples in a later email.)


Nevertheless it's still crucial to keep the shell's existing coprocess fds out of new coprocesses, otherwise you easily run yourself into deadlock.


Now, if you built bash with multiple coproc support, I would have expected you could still rig this up, by doing the redirection work explicitly yourself. Something like this:

        coproc UP   { stdbuf -oL tr a-z A-Z; }
        coproc DOWN { stdbuf -oL tr A-Z a-z; }

        # make user-managed backup copies of coproc fds
        exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
        exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}

        coproc THREEWAY { tee /dev/fd/$up_w  /dev/fd/$down_w; }


But the above doesn't actually work, as it seems that the coproc shell (THREEWAY) closes specifically all the pipe fds (beyond 0,1,2), even the user-managed ones explicitly copied with exec.

As a result, you get back errors like this:

        tee: /dev/fd/11: No such file or directory
        tee: /dev/fd/13: No such file or directory


That's the case even if you do something more explicit like:

        coproc UP_AND_OUT { tee /dev/fd/99  99>&$up_w; }

the '99>&$up_w' redirection succeeds, showing that the coproc does have access to its backup fd $up_w (*), but apparently the shell closes fd 99 (as well as $up_w) before exec'ing the tee command.

Note the coproc shell only does this with pipes; it leaves other user managed fds like files or directories alone.

I have no idea why that's the case, and i wonder whether it's intentional or an oversight.


But anyway, i imagine that if one wants to use multi coproc support (which requires automatically closing the shell's coproc fds for new coprocs), and wants to set up multiple coprocs to communicate amongst themselves, then the way to go would be explicit redirections.

(But again, this requires fixing this peculiar behavior where the coproc shell closes even the user managed copies of pipe fds before exec'ing external commands.)


(*) to prove that the coproc shell does have access to $up_w, we can make a shell-only replacement for tee(1) : (actually works)

        fdtee () {
          local line fd
          while read -r line; do
            for fd; do
              printf '%s\n' "$line" >&$fd;
            done;
          done;
        }

        coproc UP   { stdbuf -oL tr a-z A-Z; }
        coproc DOWN { stdbuf -oL tr A-Z a-z; }

        # make user-managed backup copies of coproc fds
        exec {up_r}<&${UP[0]} {up_w}>&${UP[1]}
        exec {down_r}<&${DOWN[0]} {down_w}>&${DOWN[1]}

        stdout=1
        coproc THREEWAY { fdtee $stdout $up_w $down_w; }

        # save these too, for safe keeping
        exec {tee_r}<&${THREEWAY[0]} {tee_w}>&${THREEWAY[1]}


Then:  (actually works)

        $ echo 'Greetings!' >&$tee_w
        $ read -u $tee_r  plain
        $ read -u $up_r   upped
        $ read -u $down_r downed
        $ echo "[$plain] [$upped] [$downed]"
        [Greetings!] [GREETINGS!] [greetings!]


This is a pretty trivial example just to demonstrate the concept. But once you have the freedom to play with it, you find more interesting, useful applications.

Of course, for the above technique to be generally useful, external commands need access to these user-managed fds (copied with exec). (I have no idea why the coproc shell closes them.) The shell is crippled when limited to builtins.

(I'll try to tidy up some working examples with my coprocess management library this week, for the curious.)


Juicy thread hey?  I can hardly keep up!  :)

Carl



reply via email to

[Prev in Thread] Current Thread [Next in Thread]