Re: Handling files with CRLF line ending

[Greg Wooledge]

On Sat, Dec 03, 2022 at 05:40:02AM -0500, Yair Lenga wrote:
> I was recently asked to deploy a bash/python based solution to windows
> (WSL2).  The solution was developed on Linux. Bash is being used as a glue
> to connect the python based data processing (pipes, files, ...). Everything
> works as expected with a small BUT: files created by python can not be read
> by bash `read` and `readarray`.
> 
> The root cause is the CRLF line ending ("\r\n") - python on windows uses
> the platform CRLF line ending (as opposed to LF line ending for Linux).

The files can be read.  You just need to remove the CR yourself.  Probably
the *easiest* way would be to replace constructs like this:

readarray -t myarray < "$myfile"

with this:

readarray -t myarray < <(tr -d \\r < "$myfile")

And replace constructs like this:

while read -r line; do
    ...
done < "$myfile"

with either this:

while read -r line; do
    ...
done < <(tr -d \\r < "$myfile")

or this:

while read -r line; do
    line=${line%$'\r'}
    ...
done < "$myfile"

> The short term (Dirty, but very quick) solution was to add dos2unix pipe
> when reading the files.

dos2unix wants to "edit" the files in place.  It's not a filter.
I'd steer clear of dos2unix, unless that's what you truly want.  Also,
dos2unix isn't a standard utility, so it might not even be present on
the target system.