Re: bash 5.1 heredoc pipes problematic, shopt needed

[Sam Liddicott]

For the record, here is my bash workaround to convert a heredoc fd into a
tmpfile.
Code golfers - please feel free to simplify it

Just prefix the command with "herefile" and the fd that needs converting to
a file,
e.g.
  herefile 3 command ... /dev/fd/3 .... 3<<<blah

# $1 is the herefoc filedescriptor to convert to a file
# ${@:2} is the command and args to run
# @ and _ are abused as lexical scope variables
herefile() {
  set -- "$(mktemp)" "$@"
  test -n "$1" || return $?
  { set -- "$_" "$@" # save write fd _ before it gets trashed
   { set -- "$_" "$@" # save read fd _ before it gets trashed
    # $1 is fd to read tmp filename
    # $2 is fd to write tmp filename
    # $3 is now tmp filename
    # $4 is fd of data to copy to $1
    # ${@:5} is command to run

    # now it's gone and can't leak data
    rm -f "$3" &&
    # populate the tmp file from the fd
    eval 'cat <&'"$4"' >&'"$2" &&
    # run command reading from file via specific fd but closed tmp file
    eval '"${@:5}"' "$4<&$1" "$1<&-" "$2<&-"
    # remember exit code and fd's to close
    set -- "$1" "$2" "$?"
    # finally close the tmp files cos bash doesn't do it for you if you
name them
    eval exec "$1<&-" "$2<&-"
    # return saved exit code
    return "$3"
   } {_}<"$2" # open tmp file into _ and save at the top of the block
  } {_}>"$1" # open tmp file into _ and save at the top of the block
}

# tests without the herefile fix
$ python2 -c 'import os; print(os.path.isfile("/dev/fd/3"))' 3<<<x
False
$ stat -L -c '%F' /dev/fd/3 3<<<x
fifo
$ ls -l /dev/fd/ 3<<<x
total 0
lrwx------ 1 sliddicott sliddicott 64 Apr 22 16:12 0 -> /dev/pts/1
lrwx------ 1 sliddicott sliddicott 64 Apr 22 16:12 1 -> /dev/pts/1
lrwx------ 1 sliddicott sliddicott 64 Apr 22 16:12 2 -> /dev/pts/1
lr-x------ 1 sliddicott sliddicott 64 Apr 22 16:12 3 -> 'pipe:[170069]'
lr-x------ 1 sliddicott sliddicott 64 Apr 22 16:12 4 -> /proc/17483/fd

# tests with
$ herefile 3 python2 -c 'import os; print(os.path.isfile("/dev/fd/3"))'
3<<<x
True
$ herefile 3 stat -L -c '%F' /dev/fd/3 3<<<x
regular file
$ herefile 3 ls -l /dev/fd/ 3<<<x
total 0
lrwx------ 1 sliddicott sliddicott 64 Apr 22 16:05 0 -> /dev/pts/1
lrwx------ 1 sliddicott sliddicott 64 Apr 22 16:05 1 -> /dev/pts/1
lrwx------ 1 sliddicott sliddicott 64 Apr 22 16:05 2 -> /dev/pts/1
lr-x------ 1 sliddicott sliddicott 64 Apr 22 16:05 3 ->
'/tmp/tmp.rkDXYvp4hn (deleted)'
lr-x------ 1 sliddicott sliddicott 64 Apr 22 16:05 4 -> /proc/16519/fd

On Fri, 22 Apr 2022 at 14:51, Sam Liddicott <sam@liddicott.com> wrote:
>
> Configuration Information [Automatically generated, do not change]:
> Machine: x86_64
> OS: linux-gnu
> Compiler: gcc
> Compilation CFLAGS: -g -O2 -flto=auto -ffat-lto-objects -flto=auto
-ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security
-Wall
> uname output: Linux junior 5.15.0-25-generic #25-Ubuntu SMP Wed Mar 30
15:54:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
> Machine Type: x86_64-pc-linux-gnu
>
> Bash Version: 5.1
> Patch Level: 16
> Release Status: release
>
> Description:
>         Listed in the changes:
>         c. Here documents and here strings now use pipes for the expanded
>            document if it's smaller than the pipe buffer size, reverting
>            to temporary files if it's larger.
>
>         This causes problems with many programs suffering from the TOCTOU
>         bug of checking whether or not the input is actually a file
>         instead of just using it as one.
>
>         e.g. "repo" tool performs in manifest_xml.py:
>
>           if not os.path.isfile(path):
>             raise ManifestParseError('manifest %s not found' % name)
>
>         This bug is clearly in repo and (these other tools) and certainly
is
>         not bash's fault but there is going to be a lot of breakage with
the
>         short and medium term remedy to downgrade to bash 5.0
>
>         I *like* that files aren't needed any more but there are decades
>         of scripts integrating with tools than make such checks, and which
>         work on the unknown accidental assumption that heredocs are files.
>
> Repeat-By:
>         python2 -c 'import os; print(os.path.isfile("/dev/fd/3"))' 3<<<x
>
>         emits True for bash 5.0 and before but emits False for bash 5.1
>
> Fix:
>         Please could we at least have a shopt to maintain the old
behaviour?
>
>
>