Re: puzzling problem with read seemingly losing data

[Hendrik Fritsch]

Thank you so much. That was driving me nuts.

How did I not notice that. Suddenly all that output from ffmpeg stoped
too :)

On 29.12.20 02:54, Daniel Mills wrote:
> https://mywiki.wooledge.org/BashFAQ/089
>
> On Mon, Dec 28, 2020 at 8:51 PM Hendrik Fritsch <hendrik.fritsch@web.de>
> wrote:
>
>> Hello,
>>
>> I am writing with a rather puzzling problem.
>> It seems that 'read' loses some data?
>>
>> I want to process some JSON data with the following:
>>
>>> videodata="$(jq -r -c .data[] "$t/videos.json" | tee "$t/videodata")"
>>> while read -r video; do
>>>       echo ----
>>>       echo "video $video"
>>>
>>>       videoid="$(echo "$video" | jq -r .id -)"
>>>       echo "videoid $videoid"
>>>
>>>       # ... other stuff
>>>       ffmpeg -n -i "$hlsurl" -c copy "$outname"
>>>
>>> done <<<"$videodata"
>>
>> (videos.json is just one big json object, and videodata will contain one
>> object per line (verified with tee).)
>>
>> The first few iterations work just fine and 'jq' can parse '$video' to
>> get '$videoid'.
>> Until ffmpeg ran once. (It does not run, when an output file exists
>> already.)
>>
>> The problem is the first iteration AFTER FFMPEG HAS RUN ONCE. Then I get
>> the following output:
>>
>>> ----
>>> video
>> 00Z"}],"clearkeyEnabled":true,"contentDescriptors":[],"contentRatings":[],"customAttributes":{},"description":"<snip>
>>
>> There is a substantial amount of data missing at the beginning of the line.
>> In '$t/videodata' I can lookup the actual content of the line:
>>
>>>
>> {"attributes":{"airDate":"","alternateId":"","availabilityWindows":[{"package":"","playableEnd":"","playableStart":"<snip>00Z"}],"clearkeyEnabled":true,"contentDescriptors":[],"contentRatings":[],"customAttributes":{},"description":"<snip>
>>
>> I also tried this syntax: ( while...; done < <(jq ...) ) but the problem
>> is the same.
>>
>> (It might be relevant here - ffmpeg outputs enormous amounts of log data.
>> I don't know why, they seem to come from the matroska muxer(?) and look
>> like debug output.
>> '-loglevel warning' does not get rid of them.)
>>
>> Debian GNU/Linux 9.13 (stretch)
>> GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
>> Package version 4.4-5
>>
>> Thanks,
>> Hendrik
>>
>>
>