[bug #46242] Race condition when input file is updated while compiling

[Egmont Koblinger]

URL:
  <http://savannah.gnu.org/bugs/?46242>

                 Summary: Race condition when input file is updated while
compiling
                 Project: make
            Submitted by: egmont
            Submitted on: Sun 18 Oct 2015 05:28:22 PM GMT
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.0
        Operating System: Any
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

There's an inherent race condition in the way "make" handles timestamps. If
you save a newer version of a file while make is running and is just compiling
that input file, the change won't be picked up by subsequent runs of "make",
and you'll be left with an out-of-date binary.

Steps to reproduce:

1. Have some project.
2. Open one of the large source files that takes noticeable time to compile.
3. Edit this file in some valid way (changing the functionality) but do not
save it yet.
4. In a terminal, run "make" to compile your project.
5. While make is running, go back to your editor, but keep your eyes on make's
output.
6. When make launches that particular gcc (or whatever) that compiles this
large file, save the new version.
7. Go back to the terminal. make might have completed by now. If not, you
might either interrupt it, or wait for it to complete.
8. Execute make again to catch up with the edits you performed. Wait for it to
complete.
9. Launch your app, notice that it does not include your recent changes.

The behavior is obvious from the way make checks the timestamps. The source of
the file has an earlier timestamp than the compiled object file, since you
saved the file before gcc wrote out its output. But these are not
corresponding versions.

-----

I was thinking about it for a couple of days now whether it's a bug in gcc or
make. gcc could be modified to touch back the object file to the timestamp
when it started up, and that would protect against it. (It'd also need to
write a temporary file and rename in the end, just in case it's interrupted -
I'm not sure if it does this or not). However, this would then needed to be
implemented in all compilers, and even in every one-off command that people
use from Makefiles to create files (generated source code, translations
updated with msgfmt, and a whole lot more). So I think this approach is not
feasibe, and wouldn't be the right design. gcc's job is to compile, make's job
is to do the orchestration, so in my opinion this should somehow be solved in
make.

Make does obviously check the timestamp of all input files when it starts
executing a rule. Upon the rule's completion, I think it should check the
input files' timestamps again, and if any of them were updated in the mean
time, it should re-execute the given rule.

This change, on its own, would prevent the above scenario to occur as long as
you don't interrupt any of the make/gcc/whatever processes. However, a badly
timed interrupt could still leave an obsolete compiled file with a new
timestamp on the filesystem, causing subsequent makes not picking up the
change. And by this "badly timed" I mean a giant window: the entire duration
of re-running gcc or whichever compiler.

When make detects that one of the input files was altered, it could
immediately remove the relevant output file (or touch back to an older
timestamp) before re-executing the rule. This would not completely eliminate
this race window, but would cut it down to magnitudes shorter. After all, I
assume gcc exits immediately after creating its output (what else would it
do?) and make's very next steps would be to check the timestamps.

As mentioned in #46193 comment 1, ideally rules should write the output to a
temporary file and atomically rename. If this is not the private internal
business of a rule, but rather something make is in control of, the ideal
workflow would be: First run the external command (e.g. gcc) telling it
explicitly to store its output in a temporary file, then check if any of the
source timestamps changed (and rerun gcc if it did), and finally rename. This
probably couldn't be done automatically for any rule, since it'd break when
the rule spells out the dest file name rather than referring to it as 
address@hidden But
maybe it could automatically be enabled for rules that do contain $@ and this
wouldn't break too many Makefiles. Or maybe we don't need to overcomplicate
and we're okay with this tiny race window explained at the end of the previous
paragraph.

Or perhaps you have an even better idea :)




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?46242>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/