help-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

puzzling problem with read seemingly losing data


From: Hendrik Fritsch
Subject: puzzling problem with read seemingly losing data
Date: Tue, 29 Dec 2020 02:03:23 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0

Hello,

I am writing with a rather puzzling problem.
It seems that 'read' loses some data?

I want to process some JSON data with the following:

> videodata="$(jq -r -c .data[] "$t/videos.json" | tee "$t/videodata")"
> while read -r video; do
>       echo ----
>       echo "video $video"
>
>       videoid="$(echo "$video" | jq -r .id -)"
>       echo "videoid $videoid"
>
>       # ... other stuff
>       ffmpeg -n -i "$hlsurl" -c copy "$outname"
>
> done <<<"$videodata"

(videos.json is just one big json object, and videodata will contain one
object per line (verified with tee).)

The first few iterations work just fine and 'jq' can parse '$video' to
get '$videoid'.
Until ffmpeg ran once. (It does not run, when an output file exists
already.)

The problem is the first iteration AFTER FFMPEG HAS RUN ONCE. Then I get
the following output:

> ----
> video 
> 00Z"}],"clearkeyEnabled":true,"contentDescriptors":[],"contentRatings":[],"customAttributes":{},"description":"<snip>

There is a substantial amount of data missing at the beginning of the line.
In '$t/videodata' I can lookup the actual content of the line:

> {"attributes":{"airDate":"","alternateId":"","availabilityWindows":[{"package":"","playableEnd":"","playableStart":"<snip>00Z"}],"clearkeyEnabled":true,"contentDescriptors":[],"contentRatings":[],"customAttributes":{},"description":"<snip>

I also tried this syntax: ( while...; done < <(jq ...) ) but the problem
is the same.

(It might be relevant here - ffmpeg outputs enormous amounts of log data.
I don't know why, they seem to come from the matroska muxer(?) and look
like debug output.
'-loglevel warning' does not get rid of them.)

Debian GNU/Linux 9.13 (stretch)
GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
Package version 4.4-5

Thanks,
Hendrik



reply via email to

[Prev in Thread] Current Thread [Next in Thread]