Recursive invocation of variable_expand_string bug version 3.80

[Jon Haswell]

Given the two following Makefile fragments

Makefile
------------

define INCLUDE_template
ifneq ($(MAKECMDGOALS),clean)
   -include $(1)
endif
endef

PROGRAMS := a

a_ABS_OFILES := osbasic.o

$(foreach prog_file,$(PROGRAMS),$(eval $(call
INCLUDE_template,$($(prog_file)_ABS_OFILES:.o=.d))))

osbasic.d
--------------

source/lib/linux/osbasic.o source/lib/linux/osbasic.d : GNUmakefile
GNUmakefile.rules source/lib/linux/Makefile source/lib/linux/osbasic.c
osbasic.h source/lib/linux/osrep.h \
  osbasic.h ossync.h ostypes.h

when running Make 3.80 freshly compiled on a Duron process x86 machine
under Redhat 8.0 on kernel 2.4.18-17.8.0 will give the following error

make: *** virtual memory exhauster.  Stop.

The problem is in the routine variable_expand_string in how it handles
access to variable_buffer. When the problem occurs it has been called to
expand the $(eval $(call ... code above and it does this using the default
variable_buffer size of 200. When this routine is called recursively to
handle expanding the osbasic.d contents the variable_buffer has to be
expanded as the contents of osbasic.d are over 200 bytes long. This is done
by the call to realloc which moves variable_buffer. However when we return
to the outer invocation of variable_expand_string it still has its local
variable 'o', used as the next location in variable_buffer to use, set to
the old location of variable_buffer before it was expanded/moved. This then
leads to some crazy pointer arithmetic and the attempt to allocate some
very big buffers and hence the error seen.

The somewhat obvious fix for this problem would be to make the return
function from variable_buffer_output just an offset into the buffer rather
than the absolute pointer into the buffer. However this routine is used so
frequently and at times in other functions arithmetic is performed directly
on variable_buffer that this does not appear to be a simple patch and some
one more familiar with this code might be able to suggest a better
solution.

Thanks

Jon Haswell
IBM Research
Tel 408 927 2074 Tie 457 2074
Fax 408 927 3030 Tie 457 3030