autoconf 2.52 vs Solaris 2.6 AC_SYS_LARGEFILE problems

[Mark D. Baushke]

Greetings,

The OpenSSH project uses autoconf 2.52g to build its configure
scripts. However, the AC_SYS_LARGEFILE it uses causes problems when
building under Solaris 2.6.

The basic problem is that _LARGEFILE64_SOURCE needs to be defined on
Solaris 2.6 if AC_SYS_LARGEFILE ends up doing a 
'#define _FILE_OFFSET_BITS 64'

If _FILE_OFFSET_BITS == 64, then <sys/resource.h> will define a
'struct rlimit64' but NOT define a 'struct rlimit' leading to a
failure to compile ssh.c because 'struct rlimit' is not a complete
type.

When _LARGEFILE64_SOURCE is defined, it adds a #define rlimit rlimit64
so that uses of 'struct rlimit rlim;' will become 'struct rlimit64 rlim;'
by the compiler and the compilation will succeed.

The Solaris 2.6 lfcompile(5) man page provides the supported method of
compiling apps with largefile support under Solaris is to use the
getconf command.  Perhaps getconf(1) should be used to define the
correct flags (rather than guessing at which macros need to be
defined) to ensure compatibility with current and future releases of
Solaris?

On my Solaris 2.6 box, I see

% getconf LFS_CFLAGS
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
% getconf LFS_LINTFLAGS
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
% getconf LFS_LDFLAGS

% getconf LFS_LIBS

% getconf LFS64_CFLAGS
-D_LARGEFILE64_SOURCE
% getconf LFS64_LINTFLAGS
-D_LARGEFILE64_SOURCE
% getconf LFS64_LIBS

% getconf LFS64_LDFLAGS

%

Note, however, that just adding a

  #define _LARGEFILE_SOURCE 1

does NOT appear to fix the compilation of the OpenSSH sources...
so simple use of the getconf command seems to not solve the problem...

Right now I can either manually disable AC_SYS_LARGEFILE using the 
option --disable-largefile (and I would rather NOT need to do this)
or hack the config.h file to add a '#define _LARGEFILE64_SOURCE 1'
to get things working properly.

Applying the following patch to the OpenSSH acconfig.h and
configure.ac files works around the problem, but the openssh
maintainers would rather that the root problem be fixed in the
autoconf sources.

So, if someone who understands how to do it properly would adapt the
code below that I wrote for the OpenSSH source for use in the
AC_SYS_LARGEFILE macro of autoconf itself, I would appreciate it.

        Thanks,
        -- Mark

Index: acconfig.h
===================================================================
RCS file: /cvs/openssh_cvs/acconfig.h,v
retrieving revision 1.118
diff -u -r1.118 acconfig.h
--- acconfig.h  2001/10/22 00:53:59     1.118
+++ acconfig.h  2001/10/30 18:42:56
@@ -332,6 +332,9 @@
 /* Define if you want smartcard support */
 #undef SMARTCARD
 
+/* Define if _FILE_OFFSET_BITS also needs _LARGEFILE64_SOURCE defined */
+#undef _LARGEFILE64_SOURCE
+
 @BOTTOM@
 
 /* ******************* Shouldn't need to edit below this line ************** */
Index: configure.ac
===================================================================
RCS file: /cvs/openssh_cvs/configure.ac,v
retrieving revision 1.3
diff -u -r1.3 configure.ac
--- configure.ac        2001/10/27 17:45:37     1.3
+++ configure.ac        2001/10/30 18:42:57
@@ -24,6 +24,32 @@
 
 # System features
 AC_SYS_LARGEFILE
+if test "$ac_cv_sys_file_offset_bits" != no; then
+       AC_MSG_CHECKING(if _LARGEFILE64_SOURCE is needed)
+       AC_TRY_COMPILE(
+               [
+#include "confdefs.h"
+#include <sys/resource.h>
+               ],
+               [ struct rlimit rlim; ],
+               [ AC_MSG_RESULT(no) ],
+               [ 
+               AC_TRY_COMPILE(
+                       [
+#include "confdefs.h"
+#define _LARGEFILE64_SOURCE
+#include <sys/resource.h>
+                       ],
+                       [ struct rlimit rlim; ],
+                       [ 
+                               AC_MSG_RESULT(yes)
+                               AC_DEFINE(_LARGEFILE64_SOURCE)
+                       ],
+                       [ AC_MSG_RESULT(problems with struct rlimit found) ]
+               )
+               ]
+       )
+fi
 
 if test -z "$AR" ; then
        AC_MSG_ERROR([*** 'ar' missing, please install or fix your \$PATH ***])