Re: Error for large input files >2GB

[Greg Wooledge]

I don't quite understand what your error report is saying, but when I
was trying to read your script, I couldn't help noticing a few things:

On Tue, Feb 07, 2012 at 06:24:11PM +0100, Hardy Flor wrote:
> #!/bin/bash
> 
> inputfile="<large list of files>"

Actually it's just one file name.

> leer_max="               "
> for i in $(seq -f "%02g" 3 ${#leer_max}); do
>   eval "leer_$i=\"${leer_max:0:10#$i}\""
> done

Huh?

> line=0
> while read inode size dat1 dat2 filepath; do
>   let "line++"
>   echo -en "${leer_08:${#line}}$line\r"

If you're trying to do a continuously overwritten counter, you're making
it much more complicated than it needs to be.  Consider:

for ((i=7; i<=13; i++)); do
  printf "\r%-3d" $i
  sleep 1
  done
echo

You definitely don't need a whole fleet of leer_nn variables.  Even if
you actually did need that many of them, you'd be better served by making
an array instead of a bunch of separate strings.

>   if [ -r "$filepath" ]; then
>     set -- $(stat -c "%i %Y" "$filepath")

An alternative here would be:

read real_inode real_mtime < <(stat -c "%i %Y" "$filepath")

That's not a dramatic improvement, though.

>     test $inode -ne $1 && echo "Inode from \"$filepath\" is wrong"
>     let "ut_diff = $(date -d "$dat1 $dat2" +%s) - $2"
>     test $ut_diff -ne 0 -a $ut_diff -ne -3600 -a $ut_diff -ne 3600 && echo 
> "Date from \"$filepath\" faulty ($ut_diff)"

Chaining with -a in the "test" command is not supported.  Since you're
already using bash syntax (let, ${string:start:len}, etc.), you might as
well just use [[ which can actually do that:

[[ $ut_diff != 0 && $ut_diff != -3600 && $ut_diff != 3600 ]] && echo ...

If you must use test or [ in future scripts (for POSIX compatibility),
then you must use more than one of them:

test $ut_diff != 0 && test $ut_diff != -3600 && ...

>   else
>     echo "File \"$filepath\" is missing"
>   fi
> done <$inputfile

I don't see anything here which would cause the script to malfunction
after reading multiple gigabytes of input, unless your operating system
has errors when reading from a file of that size.