possible bug in Cray port -- "virtual memory exhausted"

[Ted Stern]

Hi Paul et al.,

I have a somewhat complicated makefile I use to build a library from nearly
8700 source files in a hierarchy several dozen source directories.  I've
automated the build using the new eval+foreach functionality of v3.80, but I
recently encountered a problem.

In each source file compilation, I also have cmds to create a dependency file
to be included during the next build.  This avoids having to have an extra
rule for building dependencies.  As it happens, about 7000 of these dependency
files are empty, and the remaining 1600 or so have some non-trivial
dependencies on c or fortran include files.

At the end of my makefile, I have the line

   -include $(ALL_FDEPS) $(ALL_CDEPS)

which includes the auto-generated files if they happen to exist but doesn't
care if they don't.

Possibly relevant is that I'm using this makefile on two different platforms
-- An SGI Origin running IRIX64 and a Cray SV1 running UNICOS.

What is interesting is that in 3.79-1, including all those files might have
taken a while, but the build proceeded normally.

With version 3.80, the SGI build completes succesfully.

The Cray SV1 build is different.  Note that due to the length of the build, I
run the build script that calls GNU make as a batch job, with a prescribed
memory limit of 1GB.  I include in the script some diagnostic echo commands.



      --------------------
      Beginning sv1 shared memory build:
      Starting Make on Mon Dec 16 17:24:48 CST 2002
      --------------------

      GNU Make 3.80
      Copyright (C) 2002  Free Software Foundation, Inc.
      This is free software; see the source for copying conditions.
      There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
      PARTICULAR PURPOSE.
      Reading makefiles...
      Reading makefile `Makefile'...
      Reading makefile `/PROJSOURCE/scripts/ExcludeBase.mk' (search path) (no ~ 
expansion)...
      Reading makefile `/PROJSOURCE/scripts/unicos/sv1/InlineBase.mk' (search 
path) (no ~ expansion)...
      Reading makefile `/PROJSOURCE/scripts/SrcList.mk' (search path) (no ~ 
expansion)...
      Reading makefile `/PROJSOURCE/scripts/DmpBase.mk' (search path) (no ~ 
expansion)...
      Reading makefile `MakeLongList.mk' (search path) (no ~ expansion)...
      Reading makefile `/PROJSOURCE/scripts/unicos/sv1/Config.mk' (search path) 
(no ~ expansion)...
      Reading makefile `/PROJSOURCE/obj/sv1/inline.fdeps' (search path) (don't 
care) (no ~ expansion)...
      Reading makefile `/PROJSOURCE/obj/sv1/indep/double/dfmkrd.Fdeps' (search 
path) (don't care) (no ~ expansion)...
      Reading makefile `/PROJSOURCE/obj/sv1/indep/double/dfmlrd.Fdeps' (search 
path) (don't care) (no ~ expansion)...
      Reading makefile `/PROJSOURCE/obj/sv1/indep/double/fbscrd.Fdeps' (search 
path) (don't care) (no ~ expansion)...
      
              <4291 lines deleted>
      
      Reading makefile `/PROJSOURCE/obj/sv1/indep/s/seqgp.Fdeps' (search path) 
(don't care) (no ~ expansion)...
      Reading makefile `/PROJSOURCE/obj/sv1/indep/s/seqio.Fdeps' (search path) 
(don't care) (no ~ expansion)...
      make: *** virtual memory exhausted.  Stop.
      
        Error in build, exiting script

As you can see, make gets about 4300 files into reading the dependencies, then
chokes due to lack of memory.

I suspect the error results from the memory cap I put on the build script, but
it is odd that including 3000 empty files should exhaust the available
memory.  On the SGI Origin, I don't impose a memory limit, so it's possible
that the same memory hogging exists but I don't notice it.

Once I deleted the zero-sized dependency files, the problem went away.  Most
likely this is because I then have to include only 1700 files and don't get
close to the 4300 limit.

I've changed my auto-dependency generation commands to delete the deps files
if they're empty, which eliminates the problem for now, but I was wondering if
some inefficient memory operations have crept into v3.80's include file
logic.  Or perhaps the eval+foreach functionality takes a lot more memory than
in my 3.79.1 version?

Ted
-- 
 Ted Stern                           Engineering Applications
 Cray Inc.                               office: 206-701-2182
 411 First Avenue South, Suite 600         cell: 206-383-1049
 Seattle, WA 98104-2860                     FAX: 206-701-2500
 Debuggers' motto:  Frango ut patefaciam -- I break in order to reveal