[bug #45244] memory consumption increases with runtime

[anonymous]

URL:
  <http://savannah.gnu.org/bugs/?45244>

                 Summary: memory consumption increases with runtime
                 Project: make
            Submitted by: None
            Submitted on: Tue 02 Jun 2015 09:37:32 PM UTC
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.1
        Operating System: POSIX-Based
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

While executing, make allocates memory to hold the automatic variables for
each target.  This memory is not free'd after the target is built, so it just
accumulates until exit.

In my use case, the variables $+, $^, and $? are responsible for most of the
memory allocation because they list all of the target's prerequisites.  My use
case has the triple-whammy of long prerequisite names and many prerequisites
on my many targets.

To reproduce the issue, please find the script "makefile_generator.rb"
attached.  This creates an example makefile to demonstrate the problem.

Commands:
makefile_generator.rb 250 300 .1 (creates 300 targets with names 250
characters long, each does "sleep .1")
valgrind --tool=massif make all
ms_print massif.out.*

I've also attached  massif_report.txt generated using these steps. (My report
comes from make 4.0, but I've observed the problem in 4.1 as well.)

The graph obviously shows the memory consumption increasing throughout the
run.

In the final sample, you can see that 90.35% of the memory was allocated via
xstrdup.  Most of these allocations come from 3 places:
| | | ->29.95% (11,181,817B) 0x405A6C: set_file_variables (commands.c:231)
| | | ->29.95% (11,181,817B) 0x405C84: set_file_variables (commands.c:315)
| | | ->29.95% (11,181,817B) 0x405CC9: set_file_variables (commands.c:318)

These correspond to the places where automatic variables are created:
$+ here:
http://git.savannah.gnu.org/cgit/make.git/tree/commands.c?id=4.0#n231
$^ here:
http://git.savannah.gnu.org/cgit/make.git/tree/commands.c?id=4.0#n315
$? here:
http://git.savannah.gnu.org/cgit/make.git/tree/commands.c?id=4.0#n318

I couldn't figure out exactly where it would be safe to free this memory, but
this modification seems to work:

--- a/remake.c
+++ b/remake.c
@@ -876,6 +876,13 @@ notice_finished_file (struct file *file)
   file->command_state = cs_finished;
   file->updated = 1;
 
+  /* I'm assuming that since this file is finished, we won't need
+   * its variables anymore...*/
+  if (file->variables)
+    free_variable_set (file->variables);
+  if (file->pat_variables)
+    free_variable_set (file->pat_variables);
+
   if (touch_flag
       /* The update status will be:
            us_success   if 0 or more commands (+ or ${MAKE}) were run and
won;

I did some quick-and-dirty benchmarking, and this change doesn't seem to cost
much extra CPU time, either.

Thanks,
mblythe



    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Tue 02 Jun 2015 09:37:32 PM UTC  Name: makefile_generator.rb  Size: 374B
  By: None

<http://savannah.gnu.org/bugs/download.php?file_id=34149>
-------------------------------------------------------
Date: Tue 02 Jun 2015 09:37:32 PM UTC  Name: massif_report.txt  Size: 157kB  
By: None

<http://savannah.gnu.org/bugs/download.php?file_id=34150>

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?45244>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/