Re: puzzling problem with read seemingly losing data

[Daniel Mills]

https://mywiki.wooledge.org/BashFAQ/089

On Mon, Dec 28, 2020 at 8:51 PM Hendrik Fritsch <hendrik.fritsch@web.de>
wrote:

> Hello,
>
> I am writing with a rather puzzling problem.
> It seems that 'read' loses some data?
>
> I want to process some JSON data with the following:
>
> > videodata="$(jq -r -c .data[] "$t/videos.json" | tee "$t/videodata")"
> > while read -r video; do
> >       echo ----
> >       echo "video $video"
> >
> >       videoid="$(echo "$video" | jq -r .id -)"
> >       echo "videoid $videoid"
> >
> >       # ... other stuff
> >       ffmpeg -n -i "$hlsurl" -c copy "$outname"
> >
> > done <<<"$videodata"
>
> (videos.json is just one big json object, and videodata will contain one
> object per line (verified with tee).)
>
> The first few iterations work just fine and 'jq' can parse '$video' to
> get '$videoid'.
> Until ffmpeg ran once. (It does not run, when an output file exists
> already.)
>
> The problem is the first iteration AFTER FFMPEG HAS RUN ONCE. Then I get
> the following output:
>
> > ----
> > video
> 00Z"}],"clearkeyEnabled":true,"contentDescriptors":[],"contentRatings":[],"customAttributes":{},"description":"<snip>
>
> There is a substantial amount of data missing at the beginning of the line.
> In '$t/videodata' I can lookup the actual content of the line:
>
> >
> {"attributes":{"airDate":"","alternateId":"","availabilityWindows":[{"package":"","playableEnd":"","playableStart":"<snip>00Z"}],"clearkeyEnabled":true,"contentDescriptors":[],"contentRatings":[],"customAttributes":{},"description":"<snip>
>
> I also tried this syntax: ( while...; done < <(jq ...) ) but the problem
> is the same.
>
> (It might be relevant here - ffmpeg outputs enormous amounts of log data.
> I don't know why, they seem to come from the matroska muxer(?) and look
> like debug output.
> '-loglevel warning' does not get rid of them.)
>
> Debian GNU/Linux 9.13 (stretch)
> GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
> Package version 4.4-5
>
> Thanks,
> Hendrik
>
>