Re: weird bash pipe behavior

[Greg Wooledge]

On Thu, Jan 21, 2010 at 06:11:10AM -0800, Yan Seiner wrote:
> Greg Wooledge wrote:
> >grep definitely has this buffering behavior.  If you're using GNU grep,
> >you can give it the --line-buffered option to change this, at least
> >according to whichever contributor added that one to
> >http://mywiki.wooledge.org/BashFAQ/009
> >
> That is the correct solution.  Apparently the behavior is not limited to 
> grep; any process in a pipe that's longer than 2 can exhibit this.  
> Something about libc behavior...

It's not the pipeline; it's the command that's doing the writing which
causes the undesired behavior.

Most standard Unix commands will use large buffers when writing if
their stdout is not a terminal.  Compare:

$ for i in {1..10}; do sleep 1; echo $i; done | grep .

$ for i in {1..10}; do sleep 1; echo $i; done | grep . | cat

$ for i in {1..10}; do sleep 1; echo $i; done | while read x; do echo "$x"; 
done | cat

The first one writes a number every second.  The second one writes 10
numbers after 10 seconds.  The grep command buffered its output because
stdout wasn't a terminal in the second case -- hence the --line-buffered
workaround for GNU grep mentioned previously.

The third one writes a number every second despite being a 3-command
pipeline.  The while/read/echo loop doesn't buffer its output the way
grep does.