bug-make
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug #46242] Race condition when input file is updated while compili


From: David Boyce
Subject: Re: [bug #46242] Race condition when input file is updated while compiling
Date: Tue, 20 Oct 2015 14:03:48 -0700

This is mixing up two things. The article discusses ways of doing this
at the makefile level, i.e. without any mods needed to make per se.
There’s also a separate longstanding discussion with multiple
proposals for how to modify make to add a native checksum capability.

David

On Tue, Oct 20, 2015 at 1:03 PM, Egmont Koblinger <address@hidden> wrote:
> This sounds like an awesome idea!
>
> The article being 9 years old makes me wonder though... is it something
> that'll be done in the foreseeable future?
>
> thx,
> e.
>
> On Tue, Oct 20, 2015 at 9:57 PM, David Boyce <address@hidden>
> wrote:
>>
>> My view is that this is really a variant of the longstanding
>> discussion about getting make to not rely on timestamps at all and use
>> MD5 or similar digital signatures instead. There are a number of
>> reasons why timestamps can be insufficient beside this race condition,
>> so I think it might be better to put effort into getting that working
>> than doing more work on the timestamp implementation.
>>
>> Note that there are published mechanisms for using checksums with
>> existing make. For instance
>>
>> http://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes.
>>
>> David Boyce
>>
>> On Tue, Oct 20, 2015 at 11:36 AM, Egmont Koblinger <address@hidden>
>> wrote:
>> > Hi,
>> >
>> > Let me tell 2 real-life examples:
>> >
>> > 1. The one that made me find this issue:
>> >
>> > My hobby project takes 45 seconds to compile, out of which the biggest
>> > .cc
>> > file that I modify most of the time is 15 seconds.  In order to
>> > understand
>> > what's going on (it was not written by me, I'm just fixing bugs and
>> > adding
>> > new features) I often insert statements like printf("A\n"); so that I'll
>> > see
>> > the code is there.  I copy-paste it to multiple places, replacing the
>> > letter
>> > with B, C etc.
>> >
>> > Once I forgot to replace the letter. Saved, launched make, and
>> > immediately
>> > realized this mistake.  So went back, fixed the file, saved, and
>> > re-launched
>> > make (probably just pressed the Up arrow and Enter, not even caring
>> > whether
>> > the previous make had already finished or not).  I got hit by this race,
>> > and
>> > seeing only A's in the program's output mislead me into incorrectly
>> > thinking
>> > that B is not reached in the code, wasting a lot of time.
>> >
>> > I prefer uninterrupted runs of development with full focus for maybe an
>> > hour
>> > or two at a time, re-compiling the app every minute or two in certain
>> > phases
>> > of development.  Standing up and grabbing a coffee is not a viable
>> > option.
>> > (And yes, I do know about gdb, I just prefer printf-debugging whenever I
>> > can.)
>> >
>> > 2. Another example that also happens to me relatively frequently:
>> >
>> > I download a mid-size piece of free software (let's say it compiles in
>> > 10
>> > minutes) and apply a couple of patches to it.  Let's assume applying the
>> > patches takes 5 minutes (I have to figure out the order, whether they're
>> > -p0
>> > or -p1, some of them slightly conflict and I have to manually merge).
>> > If I
>> > perform these two steps sequentially, it's a total of 15 minutes.  So,
>> > instead, what I do is:  Start a make (or "make -k" preferably), and
>> > immediately afterwards start applying the patches.  This way we use the
>> > CPU
>> > time and my time in parallel.  Finally run "make" once again to catch
>> > up.
>> > This can reduce the total time in this example from 15 minutes to hardly
>> > more than 10 minutes.  Alas, chances that I'll be bitten by this race
>> > condition is not negligible.
>> >
>> > ---
>> >
>> >> I don't think make should be worried about this potential race; it's an
>> >> obvious case of user error with an easy discipline for the user to
>> >> apply
>> >> to avoid any problems.
>> >
>> > Is it really obvious?  Is it documented somewhere?  Do you really think
>> > pretty much all users of make think about it?  I, for one, first used
>> > make
>> > (not just typed the command, but understood its basics, wrote a tiny
>> > Makefile for a tiny project) about 18-20 years ago, yet this race
>> > condition
>> > never occurred to me until recently.
>> >
>> > Many years ago I was wondering what "make" would do if the source and
>> > destination files had the exact timestamp.  I had better things to do
>> > than
>> > actually try it out or look it up, I just assumed that developers had
>> > put
>> > proper thought to it and found the solution that Just Works™.  Nowadays
>> > with
>> > nanosecond timestamps I just don't care anymore.
>> >
>> > Had I thought about it, I probably would have just guessed that I'm not
>> > the
>> > first one with this race condition problem, would have assumed that
>> > there's
>> > already a clever solution for this in place which, again, Just Works™.
>> > Seeing that there isn't any, I'm kindly asking you to think about it and
>> > come up with one.
>> >
>> >> It's a rough-and-ready tool for developers, with
>> >> no pretenses of being intended for use by those who don't think about
>> >> how their tools work.
>> >
>> > Do you really claim I should deeply think about all corner cases of the
>> > tools before using them?  Given that "make" is just a tiny little
>> > fraction
>> > of all the tools I'm using, it's practically impossible.  In fact, I
>> > think
>> > it's quite the opposite!  The creators of such tools should do their
>> > best to
>> > make sure that the tools work reliably as expected.
>> >
>> > I didn't think about the internals of make -- why should have I?  I knew
>> > the
>> > goal it was serving: to create an up-to-date build in as little overall
>> > time
>> > as possible.  With my particular development pattern it fails to achieve
>> > this goal.  I don't think it's me who should've thought of this, nor
>> > that
>> > it's me who should change my workflow (touch the file after make
>> > completes,
>> > or not save while make is running, or start drinking coffee, or accept
>> > that
>> > I can not parallelize my time with the CPU's time, effectively taking
>> > away
>> > long minutes of my life many times for no good reason).
>> >
>> > In my firm opinion, it should be make's job to do whatever it can to
>> > produce
>> > an up-to-date build under any circumstance.  Hey, that's what make is
>> > all
>> > about!
>> >
>> >> Of course, if a file-system had a separate "up to date at" file-stamp,
>> >> that make can set (to the newest dependency's time-stamp) and later
>> >> read, or if make were to store this same datum in a separate database,
>> >> it would solve your problem.  It would also make it possible to have
>> >> make rules that tentatively build the new file, compare to existing,
>> >> replace if (meaningfully) different but only revise "up to date at"
>> >> otherwise.
>> >
>> > Such a tiny "database" could indeed solve this problem without race
>> > condition.  I would be more than happy to see this implemented!  It
>> > wouldn't
>> > have to be a "database", just a simple ephemeral text file that's
>> > removed on
>> > clean exit of make, listing all the currently running rules that'd have
>> > to
>> > be rerun if this file exists on startup (meaning that the previous run
>> > was
>> > interrupted or crashed).
>> >
>> >> As you note, every command that might be used in a make rule would need
>> >> to do this "right", which is a prohibitive requirement.
>> >>
>> >> Requiring every command that could be used as a make rule to abide by
>> >> this is, again, prohibitive.  [...]
>> >
>> > Some of these ideas I've thrown in might not work in practice, I agree.
>> >
>> > Please note that I had a simple propsal: re-check the timestamps of
>> > input
>> > files when a rule completes and re-run (moving away or dating back the
>> > previous output file first) if any of them changed.  This would not
>> > elimite
>> > the race condition completely, but would make the window magnitudes
>> > smaller
>> > (e.g. fraction of a millisecond instead of 15 seconds), and as such,
>> > would
>> > already be a great improvement!
>> >
>> >
>> > cheers,
>> > egmont
>> >
>> >
>> > _______________________________________________
>> > Bug-make mailing list
>> > address@hidden
>> > https://lists.gnu.org/mailman/listinfo/bug-make
>> >
>
>



reply via email to

[Prev in Thread] Current Thread [Next in Thread]