Intermittent parallel make rebuild failures

[David Highley]

We have a build process that had no issues in the past when we were on
Red Hat 3 and an older version of Cygwin. Now our core library builds
fail to link intermittently on a rebuild when building on Red Hat 5.1
and Windows using Cygwin make and Windows compilers. I did some digging
into it today and find that it is failing about half the time. I copied
code out of ClearCase and also moved it to the server hard drive to
eliminate networking and possible cache interference. For Linux the GNU
make is version 3.81-1. Build works every time if we do not parallel
build but it would take for ever to build single threaded.

When it fails if you re-run make it does no work except to link the
library. So it appears from running make with the debug option that the
top level make does not see the object file created by the lower level
make in the time when it checks the dependencies for the library. I did
more checking with strace and see only one fstat for the library when
the build fails to link and two when it succeeds.

I see a few bug reports about parallel builds failing and no resolution.

Test scenario:
- touch ../src/alarmSvc/AlarmPublisherImpl.cpp
- make -j 16 DEPFILES= DEBUG= 
MC_INTERFACES_DIR=/view/cm_mc_us_v10.=1_07/vobtags/mc_interfaces 
>linux_make.log 2>&1

Reducing the build process to essential targets and macros. The
mkcommondefs.mk file is large so I extracted the macros involved.

#-------------------------------------------------------------
Macros from mkcommondefs.mk file:
LIBDBG                  = _$(LINK_TYPE)
OBJDIR                  = $(OSTYPE)$(LIBDBG)
LIBPATH                 = $(TOP)/cpp/lib/$(OBJDIR)
LIBRARY                 = $(LIBPATH)/$(LIB_PREFIX)$(PROJECT).$(LIB_SO)
# Directory list of source code locations.
DIRS                    = \
                          $(RELSRCPATH)/alarmSvc \
                          $(RELSRCPATH)/logSvc \
                          $(RELSRCPATH)/msgSvc \
                          $(RELSRCPATH)/$(MSGSVCCOTSDIR) \
                          $(RELSRCPATH)/msgSvcSockMux \
                          $(RELSRCPATH)/recordSvc \
                          $(RELSRCPATH)/timeSvc \
                          $(RELSRCPATH)/utilitySvc

FIND_SRCS               = $(filter-out $(EXCLUDES), $(wildcard $(DIR)/*.cpp))
FIND_OBJS               = $(filter-out $(EXCLUDES), $(patsubst 
$(DIR)/%.cpp,$(DIR)/$(OBJDIR)/%.$(OBJ_EXT),$(FIND_SRCS)))
INF_DIST_SVCS_OBJS      = $(foreach DIR,$(DIRS),$(FIND_OBJS))

.SUFFIXES:


#-------------------------------------------------------------
Top level makefile:

TOP                 = ../..

PROJECT             = infdistsvcs

# Create archive library rule.
$(LIBRARY): $(INF_DIST_SVCS_OBJS)
        $(SET_E); $(MKDIR) $(LIBPATH); $(RM) $@; $(LINK)

$(INF_DIST_SVCS_OBJS): $(DIRS)

# Build library source code.
$(DIRS):
        @$(SET_E); cd $@ && PATH=$(PATH) "$(MAKE)"

# Must be located after all macro dependency targets. Only
# dependencies that are defined in a macro cause an order issue.
# The dependency macro must also be a dependency for another target.
.PHONY: $(DIRS) tests $(TDIRS) clean libclean distclean \
    checkout uncheckout docs web help gen_mcis build_mcis actuate

#-------------------------------------------------------------
Leaf makefile where source code is located:

# Navigate up to the parent "cpp", or "java", or "whatever" directory.
TOP        = ../../..

include      $(TOP)/cpp/bld/mkcommondefs.mk

# Here is the opportunity for the designer of the local subproject
# to add any def's for the local subproject.
#EXTRA_INCS =
#EXTRA_LIBS =

# Include common build rules share by build process.
include      $(TOP)/cpp/bld/mkcommonrules.mk

# Here is the opportunity for the designer of the local subproject
# to add any rules needed by the local subproject.

#-------------------------------------------------------------
Rules file mkcommonrules.mk:

# Default rule
all: rm_version $(OBJS)

# Create library object files, $(OBJS), rule.
$(OBJDIR)/%.$(OBJ_EXT): %.cpp
        $(SET_E); $(MKDIR) $(OBJDIR); $(COMPILE)

.PHONY: all rm_version clean libclean distclean updatelibs help

Note: The target rm_version is not pertinent to rebuild issue.