GOP-PROP 4: lessons from 2.14 (probable decision)

[Graham Percival]

http://lilypond.org/~graham/gop/gop_4.html

I made some quick graphs, so I recommend looking online.


** Proposal summary

What went well, what went badly? This is a discussion only; it
will be summarized, and we will refer back to it in future policy
decisions, but no new policies will be decided in this round.

We’ll have (at least) two sections: one for facts that anybody
considers relevant, and one for thoughts and commentary.


** History

A brief history of releases:
date (YYYY-MM-DD)       version comment
2008-10-28      2.11.63 nobody checking regtests
2008-11-17      2.11.64
2008-11-29      2.11.65
2008-12-23      2.12.0
2009-01-01              somewhere around here, Graham becomes
officially release manager, but Han-Wen still builds the actual
releases
2009-01-01      2.12.1
2009-01-25      2.12.2
2009-02-28      2.13.0
2009-06-01      2.13.1  note jump in time!
2009-06-27      2.13.2  first Graham release?
2009-07-03      2.13.3
2009-09-09              Graham arrives in Glasgow, gets a powerful
desktop computer, and begins serious work on GUB (sending bug
reports to Jan). It takes approximately 100 hours until GUB is
stable enough to make regular releases.
2009-09-24      2.13.4
2009-10-02      2.13.5
2009-10-22      2.13.6
2009-11-05      2.13.7
...
2010-01-13      2.12.3
...
2010-03-19      2.13.16 Bug squad starts doing a few regtest
comparisons, but IIRC the effort dies out after a few weeks (BLUE)
...
2010-08-04      2.13.29 Phil starts checking regtests (BLUE)
...
2011-01-12      2.13.46 release candidate 1 (GREEN)
...
2011-05-30      2.13.63 release candidate 7 (GREEN)
2011-06-06      2.14.0

  bugs-2.13-visualization.png
  zoom-2.13-visualization.png


** Carl’s analysis of the bugs

  lilypond-issues-analysis.csv
  lilypond-issues-analysis-trim-duplicates.csv

There 148 issues marked with Priority=Critical in the tracker.

I’ve done an analysis, and it looks to me like there was initially
a backlog of critical issues that weren’t fixed, and little work
was being done to eliminate critical issues.

Somewhere about 2010-08-01, critical issues started to disappear,
but occasional new ones appeared.

There were a couple of major changes that introduced unanticipated
regressions (new spacing code, beam collision avoidance). These
produced more than the expected number of regressions.

It appears to me that we didn’t really get serious about
eliminating critical bugs until about 2010-06-15 or so. After that
point, the number of critical bugs more-or-less steadily decreased
until we got to a release candidate.

Of particular interest, the first release candidate of 2.14 was
released on 2011-01-12. Over the next 10 days, about a dozen bugs
were reported and fixed. Release candidate 2 came out on
2011-02-09. No surge of bugs occurred with this release. Candidate
3 came out on 2011-03-13; we got 2 bugs per week. Candidate 4 came
out on 2011-03-29; 2 new bugs. Candidate 6 came out on 2011-04-07.
We got a couple of bugs per week.


** Notes, commentary, and opinions

Han-Wen: “Overall, I think this cycle took too long”
Mike: I agree
Graham: +1


Cheers,
- Graham