Re: Generating code coverage reports

[Simon Josefsson]

address@hidden (Ludovic Courtès) writes:

> Hi,
>
> Simon Josefsson <address@hidden> writes:
>
>> build-coverage:
>>      make CFLAGS=$(COVERAGE_OPTS) CXXFLAGS=$(COVERAGE_OPTS) VALGRIND=
>>      make CFLAGS=$(COVERAGE_OPTS) CXXFLAGS=$(COVERAGE_OPTS) VALGRIND= check
>>      mkdir -p doc/coverage
>>      lcov --directory . --output-file doc/coverage/$(PACKAGE).info --capture
>
> I'm not sure this is correct, because the `.gcda' files that are used as
> input in the end do not summarize the code covered by every single test.
> Concretely, this means that lcov's output slightly underestimates code
> coverage.
>
> Let me explain this.  Suppose you have tests T1 and T2:
>
>   * T1 uses code from foo.c and t1.c;
>   * T2 uses code from foo.c and t2.c.
>
> Running "make check" runs T1, then T2:
>
>   * T1 is run, producing foo.gcda and t1.gcda;
>   * T2 is run, *overriding* foo.gcda and producing t2.gcda.
>
> In the end, `lcov' is fed with foo.gcda from the T2 run, whereas what we
> really want to have is the code covered in foo.c by T1 *and* T2.
>
> Thus, after each test run, all GCDA files produced should be fed to
> `lcov' (thereby producing a `.info' file) and removed; once all tests
> run, all `.info' files should be aggregated (using "lcov -a"), and
> finally fed to `genhtml'.
>
> Does that make sense?

Oh, I am not that familiar with how the profiling code actually works,
thanks for explaining.  I haven't found anything in gnupdf that works
around the problem, so maybe gnupdf is affected as well, but I am not
certain.

Another solution is to require maintainers to aggregate all self-tests
into one big application, and profile that application.  This may be
simpler, and the application may be useful for other quality tools as
well.

/Simon