fseeko issues in mys2/mingw64

[John Donoghue]

This is a question/perhaps bug on bigfile support with fseeko in gnulib 
when being used in MSYS2/mingw64.

As a quick overview:

I have a demo program that opens a 5 Gig file add just does a seeko to 
the end of the file and then does a ftellp to get the end position.

I am importing ftello and fseeko and fflush and it replaces the fseeko 
and lseek functions which then in the depths in the gnulib calls _lseek 
rather than _lseeki64.


As an additional note, it doesnt seem to replace the lseek if we arent 
importing fflush

--------------------------------------------------
main.cpp:

#include <config.h>

#include <stdio.h>
#include <stdint.h>

int main()
{
  fprintf(stderr, "test big file\n");

  /* create 5G file if not there yet */
  FILE *f1 = fopen("big.txt", "r");
  if(!f1)
  {
   fprintf(stderr, "creating big file ...");
   fflush(stderr);
   f1 = fopen("big.txt", "w");
   if(f1)
   {
     char buf[1024];

     int64_t sz = 5LL*1024*1024*1024;

     while(sz > 0)
     {
       fwrite(buf, sizeof(buf), 1, f1);
       sz -= sizeof(buf);
     }

   }
   fprintf(stderr, "done\n");
  }
  fclose(f1);

  /* open file and test */
  FILE *f = fopen("big.txt", "r");

  fprintf(stderr, "sizeof(off_t)=%d\n",(int)sizeof(off_t));

  if(f)
  {
    int status = fseeko(f, 0, SEEK_END);
    fprintf(stderr, "seek status=%d\n", status);

    off_t p = ftello(f);
    fprintf(stderr, "tell pos=%lld\n", (long long)p);
    fclose(f);
  }

  return 0;
}

configure.ac:

AC_INIT([bigfile], [1.0.0])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])

AC_CONFIG_HEADERS([config.h])
AC_CONFIG_MACRO_DIRS([m4])

# Checks for programs.
AC_PROG_CXX
AC_PROG_CC
AC_PROG_CC_STDC
gl_EARLY

# For gnulib.
gl_INIT

AC_CONFIG_FILES([lib/Makefile Makefile])
AC_OUTPUT

-------------------------------------------------
Makefile.am:

ACLOCAL_AMFLAGS = -I m4
SUBDIRS = lib

AM_CPPFLAGS = -I$(top_builddir)/lib -I$(top_srcdir)/lib
EXTRA_DIST = m4/gnulib-cache.m4

bin_PROGRAMS = bigtest
bigtest_SOURCES = main.cpp
bigtest_LDADD = lib/libgnu.a

--------------------------------------------------
bootstrap.sh:

if ! [ -d gnulib ]; then \
  git clone git://git.sv.gnu.org/gnulib.git gnulib
fi

./gnulib/gnulib-tool  --import fflush fseeko ftello largefile stdio

touch NEWS README AUTHORS ChangeLog

autoreconf --install

--------------------------------------------------
Running bootsrap shows the actual modules imported as:

Module list with included dependencies (indented):
    absolute-header
    errno
    extensions
    extern-inline
  fflush
    fpurge
    freading
    fseek
  fseeko
    fstat
    ftell
  ftello
    include_next
    intprops
  largefile
    lseek
    msvc-inval
    msvc-nothrow
    pathmax
    snippet/arg-nonnull
    snippet/c++defs
    snippet/unused-parameter
    snippet/warn-on-use
    ssize_t
    stat-time
    stdbool
    stddef
  stdio
    sys_stat
    sys_types
    time
    unistd
    verify

--------------------------------------------------

Running native configure and make in linux builds ok.

The following modules are added to libnu.a:
  freading.o stat-time.o unistd.o fflush.o fpurge.o fseek.o fseeko.o

Running the app:

./bigtest
test big file
sizeof(off_t)=8
seek status=0
tell pos=5368709120

Running in debug, it has replaced fseek with the gnulib fseeko and calls 
lseek at line 117 of fseeko.c

------------------------------------------------
Running in a native configure in msys2/mingw64 and make builds ok;


The following modules are added to libnu.a:
freading.o stat-time.o unistd.o fflush.o fpurge.o fseek.o fseeko.o 
fstat.o lseek.o msvc-inval.o msvc-nothrow.o stat-w32.o

Running the app:

./bigtest.exe
test big file
creating big file ...done
sizeof(off_t)=8
seek status=-1
tell pos=0

Running in debug, and doing a break on _lseek

(gdb) bt
#0  0x00007ffd2f7ad1cb in msvcrt!_lseek () from 
C:\WINDOWS\System32\msvcrt.dll
#1  0x00000000004016e7 in rpl_lseek (fd=3, offset=offset@entry=0,
    whence=whence@entry=1) at lseek.c:69
#2  0x0000000000401636 in rpl_fseeko (
    fp=fp@entry=0x7ffd2f81fa90 <msvcrt!_iob+144>, offset=offset@entry=0,
    whence=whence@entry=2) at fseeko.c:45
#3  0x000000000040838c in main () at main.cpp:41

And for the lseek call that fails:

(gdb) bt
#0  0x00007ffd2f7ad1cb in msvcrt!_lseek () from 
C:\WINDOWS\System32\msvcrt.dll
#1  0x00000000004016e7 in rpl_lseek (fd=3, offset=offset@entry=0,
    whence=whence@entry=2) at lseek.c:69
#2  0x0000000000401683 in rpl_fseeko (
    fp=fp@entry=0x7ffd2f81fa90 <msvcrt!_iob+144>, offset=offset@entry=0,
    whence=whence@entry=2) at fseeko.c:117
#3  0x000000000040838c in main () at main.cpp:41

------------------------------------------------

configure output:
.
checking build system type... x86_64-w64-mingw32
checking host system type... x86_64-w64-mingw32
checking for _LARGEFILE_SOURCE value needed for large files... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... 64
checking whether the preprocessor supports include_next... yes
.
checking for 64-bit off_t... yes
checking for 64-bit st_size... no
checking whether fseeko is declared... yes
checking for fseeko... yes
checking whether fflush works on input streams... no
checking whether stat file-mode macros are broken... no
checking for nlink_t... no
checking whether ftello is declared... yes
checking for ftello... yes
checking whether ftello works... yes
.
checking whether fflush works on input streams... (cached) no
checking whether fpurge is declared... no
checking for fseeko... (cached) yes
checking whether fflush works on input streams... (cached) no
checking for _fseeki64... yes
checking whether _fseeki64 is declared... yes
checking for ftello... (cached) yes
checking whether ftello works... (cached) yes
checking whether lseek detects pipes... no
.

------------------------------------------------

Looking at the sources for gnulib lseek.c:

It is entering the lseek call which is really _lseek (parameters that 
are long) rather than 64bit instead of using the _lseeki64 call.

#if _GL_WINDOWS_64_BIT_OFF_T
  return _lseeki64 (fd, offset, whence);
#else
  return lseek (fd, offset, whence);
#endif


Looking for _GL_WINDOWS_64_BIT_OFF_T, it only gets set in 
lib/sys_types.in.h if WINDOWS_64_BIT_OFF_T is set.

Should lseek.c  call _lseeki64 regardless of _GL_WINDOWS_64_BIT_OFF_T 
value if _lseeki64 is available ? Or perhaps if (_FILE_OFFSET_BITS == 64 
&& HAVE_DECL__FSEEKI64 == 1)