reformatted: RE Build Processes

[Teala Spitzbarth]

sorry about the formatting trash
I'll try again....

Hi Bill,

Yes these are the kind of questions that are close to my heart too.
I found that Fogel's book, while very good, didn't outline much 
about 'release' strategies with CVS, and in any case Open Source 
strategies can have different priorities than commercial software 
companies.  I've set up a basic release strategy at my current 
company - though I know the real test of it is going to happen when 
we formally FCS and then have multiple lines of maintenance/patch
development along with the next revision development.

Hope the 'essay' is useful!
Cheers,
Teala

-----------------------------------------------------------------------
We use a 'build level' approach to development.
The basic process we are using is as follows:

- Nightly builds involve using cvs rtag to tag the module with a
  descriptive bl_projectDate_time tag:
                bl5_project14Aug01_0000
- The Build team has control of CVS tag name space and branching.
- Part of the reason we always do formal builds pulled from a tag 
  is that we use the $Name$ keyword in ident commands to allow for
  identification of key binaries.

  *Gotcha*: using $Name$ means that cvs will experience 'merge
   conflicts' on doing updates in any working copies pulled from 
   a tag - so your source tree was pulled from a
   bl5_project14Aug01_0000  tag, and then you run a build - 
   get a failure and try to go do a "cvs up -A" in your build tree 
   to get freshly delta's fixes, you will get merge conflicts on 
   all files that have the $Name keyword. Requires manual clean up. 
 
- When one of those nightly builds (or requested build during the 
  day)is deemed to pass acceptance criteria and enters QA, it is 
  retagged with a formal build level tag, i.e.
                PROJECT_BL5
- There is a short period during the integration cycle trying to 
  clear QA acceptance  criteria, where we 'close the tree' on an
  honor basis to new development, (i.e.until we know we have 
  reached a point that is sane enough to create a fixes branch).
- A quick diff -r is done against a pull of the official PROJECT_BL
  tag against the build tree that produced the CD release to QA, 
  to ensure the correct files got the final tag.
- Archival of the old build level happens, which includes removing 
  any old unneeded'nightly' tags - to keep the cvs 'log' output 
  more manageable.
- At this point, the main trunk moves on for next build level
  development
- If fixes are required for QA to complete their testing of say,
  PROJECT_BL5, then I start a PROJECT_BL5-fixes branch rooted off 
  the PROJECT_BL5 tag
- Within our company, developers are required to double-delta 
  changeson the fixes branch to the main trunk as well.  For 
  simplistic changes they have been able to use the  
  cvs up -jPROJECT-BL5-fixes command, or the   
  cvs up -j"one day ago" -jPROJECT-BL5-fixes command, to do 
  merges off the fixes branch.  But some developers have had 
  problems with the merges loosing changes. We don't do
  wholesale merges of the whole module off of a fixes branch back 
  onto the main trunk. This hasn't been a major issue here, as 
  often developers don't want the exact same fix in a maintenance
  branch going into the top of tree code.
- Fixes branch builds are likewise rtagged before the source pull 
  to do the build, a minor number is rev'd to indicate how many 
  fixes 'rebuilds' have been done.... i.e.
                        PROJECT-BL5-2
- All the above tagging is automated in nightly build scripts that 
  do everything from the  rtag, pull the src,kick off builds & unit 
  test passes across multiple OS platforms, run packaging on binary
  outputs, create the CD's and post results.
- Occasionally I hand tag a few files and update them using -F flag 
  to a given PROJECT-BL tag when there is a small local rebuild 
  to correct a minor problem. In general I try to have any kind 
  of delivered release roll out of a full automated build.
- Top of tree (i.e. ongoing development) builds are run at night 
  via a cronjob. The fixes branch builds, and the 
  'integration cycle' builds just before a cut goes to QA,
  involve coordination between developers and myself (build 
  engineer). This usually involves developers stomping into my 
  office exclaiming "we need another build" :-) Alternatively
  it involves me stomping into their office exclaiming, "Is that 
  fix done yet, we need to start the build!" :-)) 
  Or it might come from our program manger after a change 
  control meeting.... 
  Unlike James Youngman's environment, I do not examine actual 
  delta's.  I am more of an automation, release process and 
  packaging engineer than an engineer code-reviewing changes.   
  I think this is entirely a cultural/job definition issue within 
  your working environment.
- I anticipate that when we FCS - our maintenance releases will be
  handled like our current BL-fixes branches. I don't know yet if 
  our subsequent projects will be new modules in the CVS tree,  
  stay on the main trunk, or move to a new repository.
          

Side tools that support the release process:

- We use cvsweb to make the tree available for graphical diff'ing
- We use perl scripts to provide web pages that display all 
  CVS changes that have gone into any particular day's build- 
  this parses cvs log outputto display a chart with the file 
  name (linked back into the cvsweb tree), revisions delta'd,
  and the commit comments (which for us have a Bugzilla Bug ID) 
  There are scripts to provide some level of linking between 
  CVS and Bugzilla, which QA and I intend to install soon.

- *Gotcha*  There are bugs in 'cvs log' that have resulted in 
  incorrect log outputfor changes between tags on branches in 
  our tree (especially files that have been removed from the tree 
  show up as having changed on the branch).  I have had some good
  emails with David Taylor about this and gotten some workarounds-
  so I can at least display similar change logs between tags on 
  our fixes branch builds.

- When we are in change control and fixes need to be approved for
  either main trunk or  branch commits - we count on the 'honor 
  system'. BugID's from the comment logs in the daily top of tree
  builds, or branch fix builds are confirmed against the approved 
  list by Build team or QA.
- We don't use the edit/unedit capacity here. Some developers use 
  watches to keep an eye on critical shared files. 
- Occasionally, if there really is a need to lock an individual 
  file we use cvs admin -Ll <ver> filename.c
- I have once locked the whole tree with cvslock, when the src tree 
  was to be frozen, in a display of authority :-)

Some Things that developers tend to forget/not use correctly:

- When they go back and forth between trees that have sticky tags, 
  i.e. a tree pulled on a given tag (to reproduce a problem) or a
  branch (to work on a fix), they claim update doesn't work, they 
  can't find top of tree files they know have been checked in.
  Or they can't commit a change. 
  Answer: cvs up -A (or if you are in a branch you have to do 
  cvs up -r "Branch")!
- They do a cvs log in a branch working copy and see head 
  information at the top and think they aren't in a correct
  branch copy. Answer: cvs status will show the sticky tag for 
  the branch more clearly
- Some developers get excited about cvs up -A and code it into 
  local utilities, then they go work on branch working copy, run 
  their magic update command and voila their changes show up in 
  top of tree instead!

Things not to try:

- Don't try and raise the revision across the whole module as part 
  of a release process It is confusing - new files won't 
  automatically go in at the higher version. Stick to tags.
- Don't run scripts that execute lots of cvs commands one after the
  other - inetd locks up the port.