MSYS2/MINGW64 large file support

[John Donoghue]

Hi,  I'm writing an app that will be reading potentially large files > 4 Gb in 
Windows compiled using msys2/mingw64 and am seeing in issue that I believe is a 
bug, hwoever, could be just Im missing something, where the function the MS CRT 
function _lseek (long type parameters) Is called rather than _lseeki64 (int64 
parameters)

Attempting to break it down into a simple form:

Running Ubuntu 18, autotool etc installed as well as the gcc-mingw-w64 cross gcc

----------------------------------------------------------
A main program:

#include <config.h>

#include <stdio.h>
#include <stdint.h>

int main()
{
  fprintf(stderr, "test big file\n");

  /* create 5G file if not there yet */
  FILE *f1 = fopen("big.txt", "r");
  if(!f1)
  {
   fprintf(stderr, "creating big file ...");
   fflush(stderr);
   f1 = fopen("big.txt", "w");
   if(f1)
   {
     char buf[1024];

     int64_t sz = 5LL*1024*1024*1024;

     while(sz > 0)
     {
       fwrite(buf, sizeof(buf), 1, f1);
       sz -= sizeof(buf);
     }

   }
   fprintf(stderr, "done\n");
  }
  fclose(f1);

  /* open file and test */
  FILE *f = fopen("big.txt", "r");

  fprintf(stderr, "sizeof(off_t)=%d\n",(int)sizeof(off_t));

  if(f)
  {
    int status = fseek(f, 0, SEEK_END);
    fprintf(stderr, "seek status=%d\n", status);

    off_t p = ftell(f);
    fprintf(stderr, "tell pos=%lld\n", (long long)p);
    fclose(f);
  }

  return 0;
}

---------------------------------------------------------
Configure file:

AC_INIT([bigfile], [1.0.0])

AM_INIT_AUTOMAKE([-Wall -Werror foreign])

AC_CONFIG_HEADERS([config.h])

AC_CONFIG_MACRO_DIRS([m4])

# Checks for programs.

AC_PROG_CXX

AC_PROG_CC
AC_PROG_CC_STDC
gl_EARLY

# For gnulib.
gl_INIT

AC_CONFIG_FILES([lib/Makefile Makefile])
AC_OUTPUT

---------------------------------------------------------------
Makefile.am:
ACLOCAL_AMFLAGS = -I m4

SUBDIRS = lib

AM_CPPFLAGS = -I$(top_builddir)/lib -I$(top_srcdir)/lib

bin_PROGRAMS = bigtest
bigtest_SOURCES = main.cpp
bigtest_LDADD = lib/libgnu.a

EXTRA_DIST = m4/gnulib-cache.m4
-----------------------------------------------------------------
boostrap.sh
if ! [ -d gnulib ]; then \
           git clone git://git.sv.gnu.org/gnulib.git gnulib
fi

./gnulib/gnulib-tool  --import fseek ftell largefile stdio

touch NEWS README AUTHORS ChangeLog

autoreconf --install

------------------------------------------------------------------

Running, bootstrap to get everything ready and then shows:

Module list with included dependencies (indented):
    absolute-header
    errno
    extensions
    extern-inline
  fseek
    fseeko
    fstat
  ftell
    ftello
    include_next
    intprops
  largefile
    lseek
    msvc-inval
    msvc-nothrow
    pathmax
    snippet/arg-nonnull
    snippet/c++defs
    snippet/warn-on-use
    ssize_t
    stat-time
    stddef
  stdio
    sys_stat
    sys_types
    time
    unistd
    verify


Running a non cross configure and make works, and running the binary:
./bigtest
test big file
sizeof(off_t)=8
seek status=0
tell pos=5368709120


Running a cross configure: ./configure  --host=x86_64-w64-mingw32 and then make 
created bigtest.exe
Copying that to a msys2/mingw64 install from yesterday and running 
./bigtest.exe fails:

./bigtest.exe
test big file
sizeof(off_t)=8
seek status=-1
tell pos=0

Running in gdb with a break at fseek:

Thread 1 hit Breakpoint 1, 0x00007ffd2f7ddfb4 in msvcrt!fseek ()
   from C:\WINDOWS\System32\msvcrt.dll

>From [1] fseek in windows uses long parameters.


If I do a make dist on my sources, copying that tarball to the msys2/mingw64 
system and then do a native configure/make before running it:
./bigtest.exe
test big file
creating big file ...done
sizeof(off_t)=8
seek status=-1
tell pos=0

Running in gdb shows in getting to msvcrt!fseek () as well

So it appears repeatable native msys2/mingw64 and cross at least.

----------------------------------------------------------------
Looking at configure --host=x86_64-w64-mingw32:

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for x86_64-w64-mingw32-strip... x86_64-w64-mingw32-strip
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for x86_64-w64-mingw32-g++... x86_64-w64-mingw32-g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.exe
checking for suffix of executables... .exe
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether x86_64-w64-mingw32-g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of x86_64-w64-mingw32-g++... gcc3
checking for x86_64-w64-mingw32-gcc... x86_64-w64-mingw32-gcc
checking whether we are using the GNU C compiler... yes
checking whether x86_64-w64-mingw32-gcc accepts -g... yes
checking for x86_64-w64-mingw32-gcc option to accept ISO C89... none needed
checking whether x86_64-w64-mingw32-gcc understands -c and -o together... yes
checking whether the compiler is clang... no
checking for compiler option needed when checking for declarations... none
checking dependency style of x86_64-w64-mingw32-gcc... gcc3
checking for x86_64-w64-mingw32-gcc option to accept ISO C99... none needed
checking for x86_64-w64-mingw32-gcc option to accept ISO Standard C... (cached) 
none needed
checking how to run the C preprocessor... x86_64-w64-mingw32-gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether _XOPEN_SOURCE should be defined... no
checking for Minix Amsterdam compiler... no
checking for x86_64-w64-mingw32-ar... x86_64-w64-mingw32-ar
checking for x86_64-w64-mingw32-ranlib... x86_64-w64-mingw32-ranlib
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-w64-mingw32
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking whether the preprocessor supports include_next... yes
checking whether system header files limit the line length... no
checking for complete errno.h... no
checking for EMULTIHOP value... no
checking for ENOLINK value... no
checking for EOVERFLOW value... yes
checking whether stdin defaults to large file offsets... yes
checking for pid_t... yes
checking for mode_t... yes
checking for 64-bit off_t... yes
checking for 64-bit st_size... no
checking whether fseeko is declared... yes
checking for fseeko... yes
checking whether stat file-mode macros are broken... no
checking for sys/stat.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for sys/time.h... yes
checking for nlink_t... no
checking whether ftello is declared... yes
checking for ftello... yes
checking whether ftello works... guessing yes
checking for _set_invalid_parameter_handler... yes
checking for wchar_t... yes
checking for C/C++ restrict keyword... __restrict
checking for struct timespec in <time.h>... yes
checking for fseeko... (cached) yes
checking for ftello... (cached) yes
checking whether ftello works... (cached) guessing yes
checking whether lseek detects pipes... no
checking for ssize_t... yes
checking for struct stat.st_atim.tv_nsec... no
checking for struct stat.st_atimespec.tv_nsec... no
checking for struct stat.st_atimensec... no
checking for struct stat.st_atim.st__tim.tv_nsec... no
checking for struct stat.st_birthtimespec.tv_nsec... no
checking for struct stat.st_birthtimensec... no
checking for struct stat.st_birthtim.tv_nsec... no
checking for good max_align_t... yes
checking whether NULL can be used in arbitrary expressions... yes
checking which flavor of printf attribute matches inttypes macros... gnu
checking for nlink_t... (cached) no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating lib/Makefile
config.status: creating Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands

------------------------------------------------------
Looking at the cross compile:

It links the following modules are libgnu.a:
x86_64-w64-mingw32-ar cr libgnu.a stat-time.o unistd.o fstat.o lseek.o 
msvc-inval.o msvc-nothrow.o stat-w32.o
And created bigtest.exe:
x86_64-w64-mingw32-g++ -DHAVE_CONFIG_H -I.  -I./lib -I./lib   -g -O2 -MT main.o 
-MD -MP -MF .deps/main.Tpo -c -o main.o main.cpp
x86_64-w64-mingw32-g++  -g -O2   -o bigtest.exe main.o lib/libgnu.a


-------------------------------------------------------------

So a bug or not doing something correctly ?

-------------------------------------------------------------

[1] 
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/fseek-fseeki64?view=vs-2019

John D