[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
while read line; do; done; cannot handle big files
From: |
schomake |
Subject: |
while read line; do; done; cannot handle big files |
Date: |
Tue, 28 Apr 2009 19:50:07 +0200 |
Configuration Information [Automatically generated, do not change]:
Machine: i686
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i686'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='i686-redhat-linux-gnu'
-DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables
uname output: Linux ai17872 2.6.22.9-61.fc6PAE #1 SMP Thu Sep 27 18:27:50 EDT
2007 i686 i686 i386 GNU/Linux
Machine Type: i686-redhat-linux-gnu
Bash Version: 3.1
Patch Level: 17
Release Status: release
Description:
[Detailed description of the problem, suggestion, or complaint.]
The script
while read line
do
echo "$line"
done < larger-than-2GB-file.txt
will fail, returning erratic (shifted content) in $line after the 2GB has
been read.
Repeat-By:
[Describe the sequence of events that causes the problem
to occur.]
The input file is: big.dat
with 4357 white-space delimited items on each text record.
The first item is a long (>80 bytes) alphanumeric ASCII string, the
remaining 4356 items are ASCII floating-point values. With exception
of the NEWLINEs (ASCII=10,dec) the file does not contain byte values
below decimal 32 or above 127, decimal.
wc big.dat
115813 504597241 2528513389 big.dat
lines items bytes
Fix:
[Description of how to fix the problem. If you don't know a
fix for the problem, don't include this section.]
Wrote a program in C to check whether the file was corrupt.
It was not.
- while read line; do; done; cannot handle big files,
schomake <=