Re: Handling files with CRLF line ending

[Yair Lenga]

Thank you for suggestions. I want to emphasize: I do not need help in
striping the CR from the input files - it's simple.

The challenge is executing a working bash/python solution from Linux on
WSL, with MINIMAL changes to the scripts.

Specifically in my case, the owners of the various modules are working in
Linux. They are research people, with no access to window dev boxes. I
would also mention: the research people have little interest in
cross-platform portability issues.

Yair

On Sat, Dec 3, 2022 at 8:44 AM Greg Wooledge <greg@wooledge.org> wrote:

> On Sat, Dec 03, 2022 at 05:40:02AM -0500, Yair Lenga wrote:
> > I was recently asked to deploy a bash/python based solution to windows
> > (WSL2).  The solution was developed on Linux. Bash is being used as a
> glue
> > to connect the python based data processing (pipes, files, ...).
> Everything
> > works as expected with a small BUT: files created by python can not be
> read
> > by bash `read` and `readarray`.
> >
> > The root cause is the CRLF line ending ("\r\n") - python on windows uses
> > the platform CRLF line ending (as opposed to LF line ending for Linux).
>
> The files can be read.  You just need to remove the CR yourself.  Probably
> the *easiest* way would be to replace constructs like this:
>
> readarray -t myarray < "$myfile"
>
> with this:
>
> readarray -t myarray < <(tr -d \\r < "$myfile")
>
> And replace constructs like this:
>
> while read -r line; do
>     ...
> done < "$myfile"
>
> with either this:
>
> while read -r line; do
>     ...
> done < <(tr -d \\r < "$myfile")
>
> or this:
>
> while read -r line; do
>     line=${line%$'\r'}
>     ...
> done < "$myfile"
>
> > The short term (Dirty, but very quick) solution was to add dos2unix pipe
> > when reading the files.
>
> dos2unix wants to "edit" the files in place.  It's not a filter.
> I'd steer clear of dos2unix, unless that's what you truly want.  Also,
> dos2unix isn't a standard utility, so it might not even be present on
> the target system.
>