Bash 2.05b - While loop terminates unexpectedly

[ktk]

Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='i686' 
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-pc-linux-gnu' 
-DCONF_VENDOR='pc' -DSHELL -DHAVE_CONFIG_H  -I.  -I. -I./include -I./lib  -g -O2
uname output: Linux ktk 2.4.24 #1 Mon Jan 5 12:39:42 EST 2004 i686 unknown
Machine Type: i686-pc-linux-gnu

Bash Version: 2.05b
Patch Level: 0
Release Status: release

Description:

        I have a bash script with a "while" loop that is terminating
        unexpectedly.  After staring at my code (and introducing debug
        statements to convince me I'm not nuts [If I am, I owe somebody
        reading bash-bug a BEvERage]) I am now pretty sure this is an odd bug
        in bash.  The super-simplified code is roughly:

        #!/bin/bash
        VAR="`cat /etc/test.data`"
        echo "Debug: VAR=${VAR}"; echo
        echo "$VAR" |\
        while read name args; do
          echo "Debug: Considering \"${name}\" with args \"$args\"."
        done

        Running the script:
        ktk:~$ ./script
        Debug: VAR=foo car
        bar cdr
        mumble cadr
        frotz cdar

        Debug: Considering "foo" with args "car"
        Debug: Considering "bar" with args "cdr"
        ktk:~$

        So, what happened to mumble and frotz?

Repeat-By:

        Darn, but it's hard to tell you how to reproduce this bug.  I write
        /tons/ of complex bash scripts (what most people do in perl, I do in
        bash) and I confess this is the first time I've ever encountered what
        looks like non-deterministic behavior.  The same scripts are run on
        different machines with different GCC/GLIBC, some with vendor-compiled
        bash, some with my own vanila compile of 2.05b+officialpatches;
        sometimes they work as expected, and sometimes they stop in the middle
        for no obvious reason.  There is no "break" nor "exit" in the while
        loop, there is no "set -e", and no code is being .sourced   :-/

        What the actual script is doing is grabbing a list of system packages,
        checking the website for each, downloading newer versions as needed,
        and then calling a system-specific compile script if the
        version-loaded != version-installed.  For example, today, the script
        downloaded newer versions of lynx 2.8.5.17, spamassassin 2.63, and
        nessus 2.0.10a.  It then compiled and installed lynx and spamassassin.
        But the script mysteriously exited before it even checked whether
        "nessus" needed compilation.

        Since it's not enormous, I'm appending the script in question in its
        entirety.  Jump right to the very end, to the very last "while" loop.
        The "Debug: considering..." line should be printed for every line in
        $PACKAGES; but in today's run of the script, after compiling lynx and
        spamassassin (about halfway through the list of 17 systems) the script
        mysteriously exited with a zero exit code, never processing any of the
        remaining 8 lines.

Fix:
        Probably mind numbingly stupid on my part; I'm sure I'll be buying
        BEvERages in quantity for this list...

        

Real-Script:

        #!/bin/bash
        #
        # Program that looks for updates to any of several packages,
        # downloads updated source, and compiles.
        #
        # First we loop over each package, downloading updates.
        # Then, we loop over the updates and compile.

        PACKDIR=/usr/local/lib/updates
        CONF=/etc/find-updates.conf
        PKGTMP="/tmp/find-updates.$$"
        export PACKDIR PKGTMP

        function cleanup () {           # If we are ^C 'ed, clean up.
          local jobspec
          rm -rf "$PKGTMP"
          jobs | sed -e 's/^\[\([0-9]*\)\].*$/\1/' |\
          while read jobspec; do kill %$jobspec; done
        }

        trap cleanup EXIT
        rm -rf $PKGTMP; mkdir -p $PKGTMP

        function isenabled () {         # isenabled <package>
          if [ ! -x "$PACKDIR/$1" ]; then return 1; fi
          if ! grep -q "^[[:space:]#;]*$1\>" /etc/find-updates.conf; then
            if [ ! -s "$CONF" ] || tail -1 "$CONF" | grep -vq '^[#;]'; then
              echo >> "$CONF"
            fi
            echo "# $1" >> "$CONF"
            return 1
          fi
          grep -q "^[[:space:]]*$1\>" "$CONF"
        }

        function usage () {
          cat <<EOF >&2
        Usage:  $0 [ -n ] [ -v ] [ -f ] [ -d ] [ package-name ]
                -n      Skips checking websites for program updates.
                -v      Be more chatty during version checks and download.
                -f      Force a rebuild of an existing package.
                -d      Debug script; prints structures as they are used.

        `basename $0` looks for, downloads and compiles updates to the following
        packages:
        EOF
          for x in $PACKDIR/*; do
            if [ ! -x "$x" ]; then continue
            elif isenabled `basename "$x"`; then echo -e "\t\t`basename "$x"`" 
>&2
            else echo -e "\t\t`basename "$x"`\t(disabled)"
            fi
          done
          exit 1
        }

        if TEMP=`getopt -o "nvfd" -n "$0" -- "$@"`; [ $? -ne 0 ]; then
          usage
        fi

        eval set -- "$TEMP"

        NOCHECK=""; VERBOSE=""; PACKAGES=""; FORCE=""; DEBUG=""
        export NOCHECK VERBOSE FORCE DEBUG
        while [ 1 ]; do
          if [ "$1" = "-n" ];   then NOCHECK=1; shift
          elif [ "$1" = "-v" ]; then VERBOSE=1; shift
          elif [ "$1" = "-f" ]; then FORCE=1;   shift
          elif [ "$1" = "-d" ]; then DEBUG=1;   shift
          elif [ "$1" = "--" ]; then shift;     break
          fi
        done

        #
        # OK, we can't pass the correct string to 'sed' that will cause it to
        # correctly skip "#;" chars within quotes but chop the rest of the
        # line off after a comment, because the processing time goes through
        # the roof.  Here's the correct 'sed' expression:
        #       s/^\(\([^#;\"]*\(\"[^\"]*\"\)\?\)*\)[#;].*\$/\1/
        # So we cheat by assuming that no arguments may contains "#" or ";".
        #
        for x; do
          if PKG="$x"; [ ! -x "$PACKDIR/$PKG" ]; then
            echo "No package named \"$PKG\" was found; skipping it."
          else
            if ! isenabled "$PKG"; then
              echo "Note: package \"$PKG\" is not enabled.  Forcing."
            fi
            ARGS="`grep "^[[:space:]]*$PKG\>" "$CONF" |\
                   sed -e "s/^[[:space:]]*$PKG[[:space:]]*//" \
                       -e 's/[[:space:]]*[#;].*$//'`"
            PACKAGES="$PACKAGES"$'\n'"$PKG $ARGS"
          fi
        done
        PACKAGES="${PACKAGES:1}"

        if [ -z "$PACKAGES" ]; then     # do them all
          PACKAGES="`grep "^[[:space:]]*[a-z]" "$CONF" |\
                     sed -e 's/[[:space:]]*[#;].*$//'`"
        fi

        if [ -n "$DEBUG" ]; then
          echo -e "\nDebug: PACKAGES:\n$PACKAGES"
        fi

        
#########################################################################
        #                                                                       
#
        #         First, run all the version checks in the background.          
#
        #         Then, we fetch all of the updated source.                     
#
        #         Finally, in the foreground, we build each new package.        
#
        #                                                                       
#
        
#########################################################################

        if [ -z "$NOCHECK" ]; then
          echo -n "Checking for updates..."
          if [ -n "$VERBOSE" ]; then echo; fi
          echo "$PACKAGES" | (
            while read PKG ARGS; do
              if [ -x "$PACKDIR/$PKG" ]; then
                "$PACKDIR/$PKG" version-available $ARGS > "$PKGTMP/$PKG" 2>&1 &
              else
                echo "Package \"$PKG\" in $CONF ?"
              fi
            done
            wait )
          if [ -z "$VERBOSE" ]; then echo; fi

          echo "$PACKAGES" | (
            FETCH=""; FAIL=""
            while read PKG ARGS; do
              if [ ! -x "$PACKDIR/$PKG" ]; then continue
              elif [ ! -s "$PKGTMP/$PKG" ]; then
                echo "No data returned from $PACKDIR/$PKG version-available"; 
continue
              elif grep -iq fail "$PKGTMP/$PKG"; then
                if [ -n "$VERBOSE" ]; then
                  echo -n "Could not fetch available version for $PKG: "
                  sed -e 's/^fail: *//' "$PKGTMP/$PKG"
                else
                  FAIL="$FAIL, $PKG"
                fi
                continue
              fi
              A="`head -1 "$PKGTMP/$PKG"`"
              L="`"$PACKDIR/$PKG" version-loaded $ARGS 2>/dev/null`"
              if [ "$A" = "$L" ]; then
                if [ -n "$VERBOSE" ]; then
                  echo "Loaded version of $PKG is up to date: $L"
                fi
                continue
              fi
              "$PACKDIR/$PKG" fetch-version "$A" $ARGS > "$PKGTMP/$PKG" 2>&1 &
              if [ -n "$VERBOSE" ]
                then echo "Fetching $PKG version \"$A\"..."
                else FETCH="$FETCH, $PKG:$A"
              fi
            done
            if [ -n "$FAIL" ]; then
              echo "Could not check for newer versions of ${FAIL:2}"
            fi
            if [ -n "$FETCH" ]; then
              echo "Fetching newer versions of ${FETCH:2}"
            fi
            wait )
        fi

        # Now go and build new versions of sources we (may have) just 
downloaded.

        if [ -n "$DEBUG" ]; then
          echo -e "\nDebug: PACKAGES to build:\n$PACKAGES"
        fi

        if echo -n "Checking installed program versions..."; [ -n "$VERBOSE" ]; 
then
          echo
        fi
        echo "$PACKAGES" |\
        while read PKG ARGS; do
          if [ -n "$DEBUG" ]; then
            echo "Debug: Considering package \"$PKG\"..."
          fi
          if [ ! -x "$PACKDIR/$PKG" ]; then
            echo -e "\nCould not execute/build \"$PACKDIR/$PKG\"."; continue
          elif [ -e "$PKGTMP/$PKG" ] && grep -q fail "$PKGTMP/$PKG" 
2>/dev/null; then
            echo -e "\nCould not fetch newer version of $PKG:"
            sed -e 's/^fail: *//' "$PKGTMP/$PKG"; continue
          elif L="`"$PACKDIR/$PKG" version-loaded $ARGS 2>&1`"; [ $? != 0 ]; 
then
            echo -e "\nCould not check version for $PKG: $L; skipping..."; 
continue
          elif [ -z "$L" ]; then
            echo -e "\nThere is no loaded version of $PKG; skipping..."; 
continue
          elif I="`"$PACKDIR/$PKG" version-installed $ARGS 2>&1`"; [ $? != 0 ]; 
then
            echo -e "\nNote: While checking installed version of $PKG: $I"
          fi
          if [ "$L" = "$I" -a -z "$FORCE" ]; then
            if [ -n "$VERBOSE" ]; then
              echo "Installed version of $PKG is up to date: $I"
            fi
            continue
          fi
          if [ -z "$CRLF" ]; then echo; CRLF=1; fi
          echo -n "Building $PKG version $L ..."
          B="`"$PACKDIR/$PKG" build-version "$L" $ARGS 2>&1`"; bx=$?
          I="`"$PACKDIR/$PKG" version-installed $ARGS 2>&1`"
          if [ $bx != 0 -o "$I" != "$L" ]; then
            echo " Fail:"; echo; echo "$B"; echo
            if [ "$I" != "$L" ]; then
              echo "$PKG new version is \"$I\", but loaded version is \"$L\"."
            fi
          else
            echo " OK"
          fi
        done
        if [ -z "$VERBOSE" ]; then echo; fi

        # End of script