[bug #44555] "make" fails to use parallelism

[anonymous]

Follow-up Comment #2, bug #44555 (project make):

I share your surprise that fork vs vfork could possibly have an important
performance impact, but
*it's what git bisect told me
*testing immediately before the commit, I did not encounter the problem
*testing at the tip of master with the commit reverted, I did not encounter
the problem

I believe I have created a standalone test which captures what is going on for
us.

"mkmk.sh" creates a lot of dummy .c and .h files and also prints a Makefile on
its stdout.  Like our software build systems, there are a lot of object files
all with a lot of dependencies. (I had originally written some paragraphs
describing our build system, but a runnable Makefile has got to be worth a
thousand words of description)

With make.git at 4.1-11-ga80a8b8, "make -j10" typically takes about 3 minutes.
 With the attached patch applied, "make -j10" typically takes about 11s.  If
the only time consumed was by 'sleep .01', at 10x parallelism, it would
ideally take 5s.  Fully serialized, it would take 500s.

Increasing NI/NJ (number of sources and number of includes) and the number of
repetitions of '@mkdir .' all seem to make the problem worse.


$ mkmk.sh > Makefile
$ rm -f *.o default
$ time make.bad -j12 ;# 4.1-11-ga80a8b8
real    2m58.167s
user    0m2.536s
sys     2m55.375s
$ rm -f *.o default
$ time make.good -j12 ;# patch applied
real    0m10.869s
user    0m2.120s
sys     0m1.564s
$ rm -f *.o default
$ time /usr/bin/make -j10 ;# debian make 3.81
real    0m10.727s
user    0m2.244s
sys     0m1.404s


I agree we're getting into weird territory here.  It could be libc or kernel
specific.  As I mentioned initially, we're on Debian 7.8.

Linux parrot 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 GNU/Linux
GNU C Library (Debian EGLIBC 2.13-38+deb7u8) stable release version 2.13, by
Roland McGrath et al.


I also tested on a digitalocean.com virtual server instance with 12 threads
and Ubuntu 14.10 (linux 3.16, glibc 2.19) and got 36s vs 6m19 wall time (also
at -j10 parallelism)

I also tested on an atlantic.net virtual server instance with 1 thread and
Debian Testing (linux 3.18, libc 2.19) and got 1m16 vs 4m52 wall time (also at
-j10 parallelism)

So whatever's going on, it affects a range of libc and kernel versions too. 
(however, I did just copy the make.good and make.bad binaries, so it could
still be something that is specific to the debian 7 build environment)

(file #33381, file #33382)
    _______________________________________________________

Additional Item Attachment:

File name: mkmk.sh                        Size:0 KB
File name: 0001-Fix-paralellism-by-reverting-to-use-of-vfork.patch Size:5 KB


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?44555>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/