[bug #40639] GNU Make with profiling information

[Eddy Petrișor]

Follow-up Comment #5, bug #40639 (project make):

Pe 25.11.2013 04:33, "Paul D. Smith" a scris:
>
> Follow-up Comment #4, bug #40639 (project make):
>
> Hi Eddy; thanks for your interest in improving GNU make!

Hi Paul,

You're welcome.

>  I think the idea of
> providing more statistics for builds is very interesting.  While I agree it
> could be done externally (for example, by setting the SHELL variable to a
> command that would track the time spent) this could also have detrimental
> impacts (for example by setting SHELL you disable the fast-path ability of
GNU
> make to directly fork/exec simple commands, without a shell).  You're also
> adding an extra fork/exec step into each command.
>
> I have a few things to mention, however.  First, the code as you have it
won't
> work on all platforms; in particular it won't work on Windows as Windows
> doesn't (by default) provide a gettimeofday() function.  There are various
> implementations around, so this isn't a huge issue.

I'll have this in mind and fix it once other details are ironed out.

>
> Second, as written your patch is large enough that applying it would
require
> copyright assignment of the changes by you to the FSF.  If you're
interested
> in pursuing this email me directly and we'll discuss the details.

I have no problem with that. OTOH, the initial work was done by somebody else
(identified himself as 'Openshift guest'/zlqhem - see my profile-rebase branch
history), so I don't know how that fares with the FSF.

>
> Third, I'm not sure about some things: for example, why use doubles when
we're
> talking about integral values?  Actually GNU make has support for building
on
> systems which don't even support floating point, so it's best to avoid
> doubles.

This was part of the initial patch and the only reason to keep double was the
precision of the time stamps calculation in milliseconds. I'll look into some
other solution during the weekend.

> Lastly, and this is where we may need to have more conversation, I'm not so
> excited about adding the formatting capability, at least not this way.  I
> think that it could be a very useful thing to allow for specially-formatted
> output from GNU make.  For example, perhaps an output format in XML that
could
> be easily sucked into tools like Eclipse or whatever for further parsing
(I'm
> not a huge fan of XML but it is relatively universal).  Now that we have
the
> output sync capability it would be straightforward to combine these and
format
> the output of recipes for proper XML encoding as well.
>
> But I don't want to add multiple different formatting options, for
different
> types of output.  I'd prefer to have one comprehensive formatting
capability.

AFAICT, we are talking about two types of formatting:
- what information to print out
- how the information is presented (plain text, XML, other file format)

I think the second part should be more about some external tools getting the
info from Make and presenting the info in other formats. So I'll focus from
now on only on the first side of the issue.

>
> So what I'd prefer is to split it so that this change deals solely with the
> profile information, and (if requested) will print it out in a standard
> (predefined) format.
>
> Then at some future time someone can look into providing an output
formatting
> definition capability for GNU make.
>
> What do you think about this idea?  It may require some post-processing via
> sed, or whatever, to get the format needed for various purposes.


The initial patch from zlqhem just printed out on stdout the target and time
stamps for start and stop. I tried to use this patch as it was, but due to the
lack of differentiating information, it was impossible. So the first thing I
changed was to enclose the target name between square brackets and prefix the
with some hard coded string ('PROFILING:').

I played with this version a little and thought this 'one size fits all'
approach was good enough.

In practice I have found the exact opposite to be practical.

Once I tried measuring a more complex build system with recursive calls, the
recursion level was important. So was the pid.

When importing the data into Calc, having the duration directly available
(without effort from the importer) was very practical.

After adding these to the hard coded information string, I realised that for
simpler build systems removing the extra info about recursion, pid and end
time stamp during import in Calc was an extra effort that would limit the
usability of the feature[1].

That's why I added the format string feature. I dislike the unused parameters
in the prof_print_* functions, too, but I wasn't able to find a cleaner
solution.

Maybe a good compromise would be to have some hard coded predefined formatting
types and one could choose between them. 

Something like these:
- short - prints target, start and duration - e.g.:
'PROFILE:foo:1384896401494:4' - equivalent of -P'PROFILE:%N:%S:%D'
- compact - all fields present separated by ':' (without extra text) -
equivalent of -P'PROFILE:%N:%L:%P:%S:%E:%D'
- long - something like the current default formatting string, but more
verbose. - equivalent of
-P"PROFILE:target=%N:level=%L:pid=%P:start=%S;end=%E;duration=%D"

This way the profiling info can scale from simple builds to complex ones,
maybe the parsing could be eliminated entirely and the prof_print_* functions
can be dropped etc.

We can set the details later, so what do you think?



[1] TBH, I found that what would have been exceptionally useful was if I could
print time stamps relative to the earliest start time stamp so this
calculation was not necessary at all in Calc. But having all the make
instances communicate and arbitrate the minimum among them while deciding what
to print seemed like too much code would be necessary and raised some
questions regarding the printing: what happens if a slow target has the
earliest start? Should the strings be stored and then recalibrated and printed
at the very end? How useful would the partial info be for long builds?


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?40639>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/