Integration of git-based release workflow into "make dist"

[Roger Leigh]

Hi folks,

There's no patch for this yet, I'm afraid.  I just wanted to throw
out some ideas I had relating to managing releases with automake,
and integrating this with a git workflow.  Sorry if it's a bit
rambling.

I've long been using automake, and I've always used "make dist[check]"
as the end part of the release process.  When using a version control
system, be it CVS, SVN or git, the following sort of workflow happens:

• tag repo
• make new checkout to get a pristine tree
• bootstrap autotools
• configure and make and make check
• make distcheck
• distribute generated release tarball

However, the "distribute release tarball" step is becoming less and
less relevant with the advent of git.

I do a lot of Debian packaging work, as well as actual software
development for Debian.  All of this work nowadays occurs in git
repositories.  For packages of software distributed by third
parties (most packages) our workflow is to have a git repository with
a minimum of two branches:

• an upstream branch - contains the contents of the release tarball
  injected into git
• a debian branch - contains the Debian packaging

There may be other branches (tracking upstream stable/development and
different Debian releases), but these are the important ones for
the context of this discussion.  We track upstream releases on one
branch, and do the packaging work on a separate branch.  [there's also
a pristine-tar branch for reconstructing perfect copies of the
release tarball from git, but that's not particularly relevant here.]

The important point to make here is that the tarball was irrelevant.
It was just an intermediate transfer format between two version
control systems (possibly even the /same/ version control system, if
not in extreme cases the exact same repository).  In an ideal world,
the upstream releases could be tracked directly within the version
control system, without any tarballs at all.

Nowadays, it's becoming increasingly common that the upstream
developers are also using git (often it's ourselves!), and it would
greatly ease collaboration and keeping up to date, as well as
simplifying our workflow, if we could skip the tarball part and just
pull the release out of the repo.


What are we missing?

The VCS does not contain exactly the same files as the release
tarball.  It does not typically contain the result of the autotools
bootstrap, and it may contain other generated files, as well as
possibly not distributing some files only needed for work in the VCS.

We need this distributed tarball content in the VCS.  Currently we
are re-importing the tarball by hand as a downstream user, but
"make dist" could do this at the upstream end.  The upstream
developer can now have two branches in their VCS:

• a working/release branch as always
• a distribution branch containing the distributed files

The downstream user can now simply track the distribution branch
with tarball distribution and re-injection being completely
skipped.


What do we gain?

At this point, we can track all changes in git:

• tag repo
• [optionally clone clean repo to be sure you get a non-crufty
  environment]
• make distcheck [enter GPG key to sign the distributed release]

  upstream release branch
  → upstream distribution branch
    → downstream packaging branch
      [ → user modifications ]
      [ → further downstream distributor changes ]

Everything can potentially be in git, and patches can be sent between
all parties with ease.


What can automake do?

This is a rough outline of what I'd like to do (unless someone beats
me to it!)

• Add a dist-git option and Makefile target.
  This will cause $distdir to be injected into git, rather than just
  calling tar as for other git targets.

• This will require some additional options in order to work correctly:
  · A branch name (the head to append the new tree to)
  · [optional] Tag name, could be a pattern such as dist/$(VERSION)
  · [optional] Flag for signing the tag or not
  . [optional] Template commit message
  These could all just be variables in the top-level Makefile.am.

Normal git usage takes place in the working tree.  Because we're
already in a working tree, we make use of a neat git feature:
using GIT_INDEX_FILE to operate with an alternative index.  This
way, we can add the entire contents of $distdir to the alternate
index, and then commit that onto a separate branch using the
low-level git core plumbing git write-tree and git commit-tree,
followed by git update-ref.

My git-fu is not yet at master level, so I'm currently working out how
to do this completely in practice.  Other tools are already doing
similar things, for example pristine-tar, so they would be useful as a
starting point.

Here's a simple test.  Note that there might be a better way to
do this.  Currently it misses the parent off the commit step
as well as for the ref update.

% make distdir
% GIT_INDEX_FILE=idx GIT_WORK_TREE=$distdir git write-tree
⇒ 4b825dc642cb6eb9a060e54bf8d69288fbee4904
%echo "Distribution of $(PACKAGE) version $(VERSION)" [|gpg --clearsign] | git 
commit-tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
⇒ 04bd8ea48e17d845d22cfcfb10bde1a4dee06caf
% git update-ref refs/heads/distribution 
04bd8ea48e17d845d22cfcfb10bde1a4dee06caf
% git tag -s ...

This could possibly be provided as a separate script, rather than
in the generated Makefile, but would probably be more flexible if
put in the Makefile.


Debian has now got a new source package format, which includes
(currently experimental) support for git as a source distribution
format.  With the above missing piece in place, it will be possible to
have all the sources in git from upstream to distributor to end user,
with an complete GPG signed history of all changes made by upstream
and the distributor (and any other distributors downstream of Debian).
Of course, the feature as proposed has many other uses than this, but
that's where I'd like to get to!



Any thoughts or comments?


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.

signature.asc
Description: Digital signature