Re: Feature request or advice on implementation for buffering command output

[L A Walsh]

On 3/27/2019 12:49 AM, Anders Brujordet wrote:
> Usecase:
> You run a command, the output is displayed in your terminal. Now that you
> see the output, you would like to grab say an ID from this output. Instead
> of coping and pasting the output, I would like to be able to do something
> like:
> # grep 'the thing that you want' <<< ${COMMAND_OUTPUT} | ./some_script
>
> Where COMMAND_OUTPUT is the last n number of output lines from the previous
> command.
>
> Now, I've looked at several ways of implementing this is Bash. They all
> break in various ways, because it usually means pushing all output to a
> file/variable and then pushing it to stdout. Would it be feasible to have
> Bash tee off anything intended for stdout to a variable?
>   
----
    Alot of what you want to do depends on how well behaved your
data is.  I.e. if your dataoutput is random binary, then
probably not.  But since you are using grep and grep is
designed to work on 'text' lines, I'm going to start with
the assumption that the output you want to buffer is
'Text' (not binary). 

    What would be wrong with doing something like:

readarray -t last10< <(COMMAND |tail -10)

That will put the last 10 lines of COMMAND_OUTPUT into
the array 'last10'.  Now you can launch another command
to process that last 10 lines.  Note that I used the '-t'
which strips off the '\n' at the end of line.  For about
50% of the uses, you might want to keep the line ending
(and not use the '-t' option).  Example.  Using 'seq'
to generate lines of output, if I use -t, I might get:

> readarray -t last10< <(seq 1 100|tail -10)
> printf "%s" "${last10[@]}"
919293949596979899100>

Which may not be what you want.  But you could keep the
newline when reading it in and it will be used on output:

> readarray last10< <(seq 1 100|tail -10)
> printf "%s" "${last10[@]}"
91
92
93
94
95
96
97
98
99
100
>

> Is there already some way of achieving this functionality?
>   
---
    Does the above method work for your data?  Remember, the above
handles a simple case of working with text lines.
> If not, could anyone give me some pointers as to where or how this could be
> implemented?
>   
----
    I'll often "buffer" output into an array as it automatically
keeps separate data items, separated, and I don't have to worry
about resplitting the data into lines.

    The main caveat about storing data in variables is that
bash is written in 'C' and uses NUL or \000 as an end-of-string
terminator, so that can end up truncating things where you wouldn't
expect or want.  If your data has chaos-causing control codes or
NULs in it, you might want to pre-encode it before feeding it
to bash.


    Of note: the last 10 items of output won't be available until
that command has terminated, so there would
be little benefit in trying to use it in a pipe.

    If you needed to store that output and pick it up later,
_one_ possibility would be to use 'declare -p ARRAYNAME' and
direct that output to a variable or a file.  Supposing you
sent the output to a variable.  To get back the original
data use "eval".  Example:To get the array
back you can use something like:

> readarray -t last10< <(seq 1 100|tail -10)
> var="$(declare -p last10)"
> unset last10
> echo -E "length of last10: ${#last10[@]}"
length of last10: 0
> eval "$var"
> echo -E "length of last10: ${#last10[@]}"
length of last10: 10
> printf "%s\n" "${last10[@]}"   # add stripped '\n' back in
91
92
93
94
95
96
97
98
99
100

As for little quirks or why I did something a certain way,
check w/the manpage.

Hope this helps...
-linda