'while read' loop performance (redirection vs pipeline)

[Terence O'Gorman]

Configuration Information:

Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -O2 -g -pipe -Wall -Werror=format-security
-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
-fstack-protector-strong -grecord-gcc-switches
-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-Wno-parentheses -Wno-format-security
uname output: Linux quietpc 5.6.13-200.fc31.x86_64 #1 SMP
              Thu May 14 23:26:14 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu

Bash Version: 5.0
Patch Level: 11
Release Status: release


Description:

The 'while read' loop exhibits very different performance depending on
whether input is via redirection or pipeline (and increasingly diverging
performance with any increased data).  Here are some results from my
machine (also observed in versions 3.0 and 4.0):

[tog@quietpc ~]$ ~/loop.bash

100,000 lines of length 4:
'<': real = 0.609, user = 0.522, sys = 0.087
'|': real = 1.000, user = 0.561, sys = 0.793

100,000 lines of length 8:
'<': real = 0.652, user = 0.567, sys = 0.084
'|': real = 1.788, user = 0.772, sys = 1.838

100,000 lines of length 16:
'<': real = 0.660, user = 0.541, sys = 0.118
'|': real = 2.900, user = 1.019, sys = 3.389

100,000 lines of length 32:
'<': real = 0.684, user = 0.576, sys = 0.107
'|': real = 5.962, user = 1.560, sys = 7.809

100,000 lines of length 64:
'<': real =  0.763, user = 0.677, sys =  0.086
'|': real = 11.602, user = 2.803, sys = 15.727

100,000 lines of length 128:
'<': real =  0.984, user = 0.874, sys =  0.109
'|': real = 22.837, user = 5.061, sys = 31.592


Repeat-By:

#!/bin/bash

count=${1:-100000}
file=$(mktemp /tmp/loop.XXX)
string="####"

TIMEFORMAT="real = %R, user = %U, sys = %S"

for loop in {1..6}
  do
    printf "\n%'d lines of length %d:\n" $count ${#string}
    yes $string | sed ${count}q >$file

    printf "'<': "; time             while read line; do :; done <$file
    printf "'|': "; time cat $file | while read line; do :; done

    string+=$string
  done

rm -f $file