You'll only catch these problems randomly with the solution outlined below but it should help flush them out. I think it would be unwise to leave this in as a permanent change to your build, but as a one-off hack for automatically discovering dependency issues it's reasonable.
If the extra time your build(s) will take to complete won't be an issue you can increase the sleep duration. That should give you a higher probability of discovering hidden dependencies.
1) Write a simple script:
#!/usr/bin/perl
# Sleeps for a random duration <= 1 second
select( undef, undef, undef, rand( 1 ) );
unshift( @ARGV, $ENV{REAL_SHELL} );
exec( @ARGV );
2) Change SHELL in your makefile(s) to use the script
3) export REAL_SHELL to the preferred shell
I used something like that but our build is rather large so it didn't flush everything out and raising the sleep duration wasn't realistic. I ended up doing this:
0) Our build doesn't use any built-in recipes. It uses the -rR flags to disable all built-ins and all recipes are explicitly specified.
1) Modified every recipe to print graphviz-formatted dependency information to a log file
In our case it was shared libraries missing dependencies; we were [unwittingly] compensating for that lack while linking the executables, so:
2) Modified the build to use the following link flags:
solaris: -zguidance=nolazyload,nodirect,nomapfile,nolibpath,notext
linux: -Wl,--unresolved-symbols=ignore-in-shared-libs -Wl,--warn-unresolved-symbols
3) Wrote a script to correlate the unresolved symbols from the logs with libraries that could satisfy that dependency then emitted that as graphviz formatted text. I think I had trouble forcing the linker not to demangle C++ symbols so I ended up also using output from "nm" to make sure I was getting exactly the right symbol.
4) Used the output from step 3 to discover un-expressed dependencies in step 2 and added them to the build. While doing this, some dependencies had to be dropped because it created a circular dependency.
5) Used the graphviz output in dot/neato & similar tools to visualize everything
6) Scripted something that could eliminate all non-circular dependencies from the output so we could narrow in on the libraries involved.
-brian