[Gnu-arch-users] oh the heck with it -- tla-1.2pre0

[Tom Lord]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


[Please forward to interested parties.]

I changed my mind and decided not to wait.  tla-1.2pre0 is now
available.

Savannah is not quite fully up yet so please consult either of:

        http://arch.quackerhead.com/~lord
        http://regexps.srparish.net/www


The basics of revision signing and revision checksum verification are
now in place.

Enclosed is a HOWTO explaining how to use the new features.

In other news, the close of 2004 has brought me some sad (hopefully
not too sad) news and (related to that) unexpected veterinary bills.
Thus, with more than customary urgency, it's time for me to pass the
hat (see the last section of the HOWTO).

- -t



* HOWTO: Archive Integrity Checking With gnu-arch 

  arch 1.2 adds some features for _archive_integrity_checking_

  Archive integrity checking helps to address three problems:

  1) The problem of silent media failures: Integrity checking makes it
     cheap and easy to notice when some part of an archive has been
     corrupted by an otherwise unnoticed media failure.

  2) The problem of unauthorized committers: Arch generally relies on
     filesystem permissions and the authentication mechanisms of its
     transports (sftp, WebDAV, ftp, or local access) to prevent
     unauthorized commits to an archive.  What happens if someone who
     is not supposed to be authorized manages to commit to an archive
     anyway?

     Integrity checking enables all commits to be cryptographically
     signed by whomever performs the commit or whomever copies that
     commit to a mirror.   When reading from an archive, arch can now
     be configured to check those signatures and fail if they are 
     not valid for a given project.

  3) The problem of cracked project hosts: Recently we've seen the
     Debian and GNU Savannah hosts cracked, as well as the CVS mirror
     of linux kernel sources kept in Bitkeeper.  After such break-ins,
     there is critical work to do to make sure that the software
     stored there has not been tampered with.  In the case of
     Savannah, this required asking project maintainers to review all
     recent commits to their CVS archives, looking for signs
     maliciousness.  In the case of the Bitkeeper CVS mirror,
     malicious tampering appears to have taken place -- but it was
     quickly automatically discovered by integrity checking.  What is
     GNU arch doing for this situation?

     The same cryptographic signing mechanism that can be used to 
     detect unauthorized commits makes it very difficult for a cracker
     to modify already committed versions -- they must at least be
     able to freshly sign the modified files with a valid key.

     But beyond that, the information arch records to detect media
     failures can be used to system administrators to detect even
     apparently validly signed changes to archives.   That information
     can also make it easier for all holders of _mirrors_ of an
     archive to compare the data they have and notice if some 
     copies have been tampered with.

  This HOWTO explains the new features in arch and how to use them
  in (what I expect will be) common scenarios:

  Contents:

    * MD5 Checksums of Revision Data

    * Signed Archives
    ** Creating a Signed Archive
    ** Signing Rules: ~/.arch-params/signing/$ARCHIVE
    ** Signature Checking Rules: ~/.arch-params/signing/$ARCHIVE.check
    ** Default Rules

    * Signing and Mirrors
    ** Dropping Signatures from Mirrors
    ** Signing Mirrors Anew
    ** Copying Signatures to Mirrors

    * Adding Checksums to an Existing Archive

    * Converting an Existing Archive from Unsigned to Signed

    * If You've Been Running Development Sources

    * Sysadmin and Project Host Considerations
    ** Agressive Protection: Fast Batch Validation
    ** Agressive Protection: Watching for Removals and Changes

    * My Archives Are Now Signed

    * My Public Key

    * How You Can Help the GNU arch Project to Continue




* MD5 Checksums of Revision Data

  As you may know, arch operations that write to an archive
  (import, commit, tag, cacherev, and uncacherev) work by 
  storing new files in the archive.   Once stored, those files
  are never modified.   Only the files stored by cacherev may
  ever be removed.

  Beginning in tla 1.2, whenever arch writes files to an archive,
  it stores a checksum file containing MD5 hashes of the contents
  of those files.

  When arch retrieves a file from an archive, it computes an MD5
  of the file it receives and compares that to the checksum file.
  If they disagree, an error is signaled and the operation is 
  aborted.

  No special action is needed to activate this feature: beginning
  in 1.2, arch will always do this.

  If you have existing archives, you may want to add checksum data for
  the revisions already there.  A technique for doing so is presented
  later in this HOWTO.



* Signed Archives

  A signed archive is one in which the MD5 checksum data is itself
  cryptographically signed.   In addition to checking MD5 sums,
  arch can be configured to verify signatures and to signal an
  error and abort if a signature is missing or invalid.


** Creating a Signed Archive

  To create a new signed archive or mirror, pass the --signed
  option to make-archive:

        % tla make-archive --signed [...other stuff...]

  It is important to make the decision to sign or not sign an
  archive _before_ any revisions have been committed.   Signing
  will only function properly if _all_ of the revisions in an 
  archive are signed.

  If you have existing unsigned (or incompletely signed) archives, you
  may want to convert them to fully signed archives.  A technique for
  doing so is presented later in this HOWTO.



** Signing Rules: ~/.arch-params/signing/$ARCHIVE

  Before you can write revisions to a signed archive, you must
  configure a _signing_rule_ that applies to that archive.

  Normally this is done by creating a file:

        ~/.arch-params/signing/$ARCHIVE

  where $ARCHIVE is the name of an archive.   Normally, that
  file should contain a single shell command which reads a
  message from its standard input and writes a signed copy of
  the message to standard output.

  A typical example of a signing rule (the contents of the above named
  file) is:

        gpg --clearsign

  Users may find it convenient to use a "key agent" for this purpose.
  For example, using quintuple-agent one might have the rule:
  
        agpg --clearsign


  It is an error to have a signing rule for an archive which is not
  signed, or to not have a rule for an archive which is signed.
  Arch will refuse to let you commit to an archive if either of 
  those errors occurs.


** Signature Checking Rules: ~/.arch-params/signing/$ARCHIVE.check

  Arch will _always_ verify checksums found in archives.
  Additionally, arch can be _configured_ to verify signatures found
  in archives.

  Signature checkins is enabled by creating a file:

        ~/.arch-params/signing/$ARCHIVE.check

  where $ARCHIVE is the name of an archive.   That file should
  again contain a single shell command, this time one which 
  reads a signed message on standard input and exits with 0 status
  for a valid signature, non-0 status for an invalid signature.

  A minimal example of a signature checking rule is:

        gpg --verify-files -


  It is an error to have a signature checking rule for an unsigned 
  archive.  If that error occurs, arch will refuse to read from 
  that archive.

  It is _not_ an error (but will generate a warning message) to _not_
  have a signature checking rule for an archive which is signed.


** Default Rules

  You can create default rules both for signature generation and
  signature checking by storing those rules in the files:

        signing rule default:
        ~/.arch-params/signing/=default


        checking rule default:
        ~/.arch-params/signing/=default.check



* Signing and Mirrors

  How do signatures work with archive mirrors?   Are signatures
  copied from an archive source to its mirrors?  Or are mirrors
  signed anew by the person creating them?  Can signatures in a 
  source simply be dropped from a mirror?

  The answer is that any of those is possible, it's up to you:


** Dropping Signatures from Mirrors

  If you do _not_ pass the --signed option to make-archive when
  creating a mirror, then the mirror will not contain signatures
  regardless of whether the mirror's source archive contains
  signatures.

  Like all archives, the mirror will contain checksums -- but not 
  signed checksums.


** Signing Mirrors Anew

  If, when you create a mirror, you pass the --signed option
  make-archive then the mirror will be a signed archive.

  You _must_ therefore have a signing rule in order to write to the
  mirror using the archive-mirror command.

  If your signing rule for a mirror is an ordinary rule, such as:

        agpg --clearsign

  then the mirror will be signed anew.   That is, the checksum files
  in the archive will bear the signature of the user who invoked
  the archive-mirror command.

  There are circumstances where it can be useful to sign mirrors anew.
  For example, let's suppose that you and 5 friends work on a project
  in a private archive which you share.  But you also want to publish
  the project by making a mirror of that archive on a public project
  host.

  It may be simplest for your project host and for users of your
  project to use just _one_ key to sign the revisions in the mirror,
  even if the 5 of you use separate keys to sign revisions in the
  private archive.

  Configuring your mirror to "sign anew", as shown above, allows you
  to use just one key for the mirror, regardless of how many keys are
  present (or if any are present) in the mirror source.


** Copying Signatures to Mirrors

  Sometimes, though, you will want to _copy_ signatures to a 
  mirror.

  As always, you must pass the --signed option to the make-archive
  command when creating the mirror.

  To copy signatures, rather than sign anew, you need a special
  signing rule.  In the file:

        ~/.arch-params/signing/$MIRROR

  where $MIRROR is archive name of the mirror, instead of a shell
  command, store the name of the archive from which signatures should
  be copied.

  For example, let's suppose that I have a private archive which
  I have registered with the name:

        address@hidden

  and a public mirror of that archive called:

        address@hidden

  To copy signatures to the mirror, in the file:

        ~/.arch-params/signing/address@hidden

  I store the single line:

        address@hidden

  Similarly, if _you_ have registered my public mirror as:

        address@hidden

  but also have a local "mirror of that mirror" called:

        address@hidden

  then in

        ~/.arch-params/signing/address@hidden

  you should store the line:

        address@hidden

  In general, the signing rule for a mirror that copies signatures
  should contain the name of the archive from which signatures are
  to be copied.
  


* Adding Checksums to an Existing Archive

  Let's suppose that you have an existing archive.   If you've begun
  using tla-1.2 then _new_ revisions in that archive will contain
  checksum records, but _old_ revisions will not.

  You can convert your old archive to have checksums for every
  revision in just a few steps:


    1. Create a new _mirror_ of your old archive.

    2. Use the archive-mirror command to copy all old revisions
       to the the mirror.

    3. _If_ your old archive is _not_ a mirror, then delete
       the file =meta-info/mirror from your new mirror.

    4. Set aside (and later remove) your _old_ archive directory,
       and replace it with the archive directory of the new mirror.

    5. Unregister the name of the mirror created in step 1.


  When old revisions are copied in step 2, checksums will be added.




* Converting an Existing Archive from Unsigned to Signed

  Let's suppose that you have an existing archive which is _not_
  a signed archive, and now you wish it were a signed archive.

  You can convert it using the the same recipe given above
  for adding checksums to an existing archive -- just be sure
  to pass the --signed option to the make-archive command in
  step 1, and to configure your ~/.arch-params/signature files
  appropriately (as described in earlier sections).



* If You've Been Running Development Sources

  A small number of users who run the latest development sources
  of tla may have what could be called "partially signed" archives:

  These are archives that contain an "=meta-info/signed-archive" file,
  and may contain _some_ signed revisions, but may also contain some
  unsigned revisions.

  Partially signed archives are no longer supported.   If you try to 
  read data from an unsigned revision in a partially signed archive,
  arch will signal an error and abort the operation.

  To convert your partially signed archive to a fully-signed archive,
  delete the =meta-info/signed-archive file, then follow the 
  instructions given above for converting an unsigned archive to a 
  signed archive.




* Sysadmin and Project Host Considerations

  What else do system administrators and project hosts need to know
  if they want to use arch's new features to aggressively protect
  "source code assets?"

  A few things:

** Aggressive Protection: Fast Batch Validation

  A brute-force way to check the validity of signatures in an 
  archive is to use the `get-patch' command to read every revision
  in the archive.

  Faster and more convenient techniques are in the works and the 
  best place to learn about their availability and status is
  on the gnu-arch-users mailing list.   Visit:

     http://www.gnu.org/software/gnu-arch/

  for links to mailing list information.
  

** Aggressive Protection: Watching for Removals and Changes

  Intrusion detection and media-checking software should also 
  watch for the removal of revisions from an archive and for
  _changes_ (even if validly signed) to the signed checksum files
  in an archive.

  This can be accomplished with a few 10 lines of shell script
  along with a facility for safely storing a master list of expected
  archive contents.   

  If you would like help with this, again, please join and consult
  with the members of the gnu-arch-users mailing list.


* My Archives Are Now Signed

  The latest archive containing GNU arch sources is now signed.
  My public key is enclosed.   The archive coordinates are:

   archive name: address@hidden

   archive locations: 
  
    http://regexps.srparish.net/{archives}/address@hidden
    http://arch.quackerhead.com/~lord/archives/address@hidden


* My Public Key

- - -----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.2.2 (FreeBSD)

mQGiBD86h28RBACKskrF15B62qDfqHaY52QrayNzrSjjQCXCco/b0ZvR+jmqdMzQ
BdrdXBPm5NSt2jKcs3aaE4JTfMlM8eNYehwPVWYu7ZPmdZbTq65BfwI23xT5pBLM
DDeIkAiiJsNbSfSxCjsfH5cX/UmyJu1cwIZkX+sEdQoHE7Jf4PAMeo9n4wCg3iC/
qEKehJjZdobTyeLweAwdoLMEAIOCcjcNQ+Xutl+BcyETshYjgpQvSMHGW2HySMlJ
SiHAk0oTHCozgwOwUk+R8f7He49kXTEtVNOkjAEbUef00V9lziyofubRNt1qu638
CFQB+JjXjtPyAEfuTdXkElWpLmqhjVqsAQRj+04HLznhtrHFE/G7oG9xe3Oq1vrf
hOosA/93yypwdoqIflqt13dpTy0MqvggskB2N43DVMEP9FrHyjaZvK90xGzn9QVu
wBxGJb2eAigtIy08/m3+FDwLPpeppuzcp1bNo8gRfEPN1inm/pfOxIH1NdXX0OIG
Zb9/zCY3stUYMQkz4eWmoqT+fc5xx61yEvtI/CcQnbj8gbhH47QXVG9tIExvcmQg
PGxvcmRAZW1mLm5ldD6IWwQTEQIAGwUCPzqHbwYLCQgHAwIDFQIDAxYCAQIeAQIX
gAAKCRBiIvi16frxKfBgAJ0daYI1wcozxL9l+f6/PBkNR0RGFQCeI9PLnu3VYW6O
iSjVt/Wf5tThTXm5Ag0EPzqIGBAIANb71iR1Ybn6Ez4N611+is1jE0ceTR3ZiY+z
2gTpw8AkFu3qAzoRCevT8lUZDEYxqT7cDkTedBISK+Dkines10Ydtp1DF1bjbX/d
hfq/WHOJduG+6ggF7StGuBgEL9ozPTy21ek5zLB+prLHR12qsCEur8enU8YbRhNO
Y6XFIJPOvboy5m0rEMWm6oEUq2G20ZaIscjDlSS2uCnZihA5o3q8WNWgAR2vz+1b
sU5j8UuyqGZcpTPfgYVFkJHTjFjaIAp/xxqytX8nSMmaImlRHTSiyICcga6ZeaQM
/1vHdAl/XkYExaAwxhGsYsGgoi5wSa9c1LDDuos0LU4riYJaTKMAAwUH/0YFyzS6
JeFbR7aiUw9mu6bYNkZ4M0t+mq6LhtNAfNUEI4oswc06BOK0aMXZRHj5+Ww+ImNS
3XyKcD/aSuG+njVRQe86Q1jfh5KjUHrh+EbxkXPgkSuz+YxD8KsCFL0Yr6Sgkt7o
HkGFNoccYBN5mzhjmZZs75rDza1Mqu9QeitbtPo4/HCzARi7LDx35A3SGfmIwMa+
SKtJ/41O3jWTbOnzPsIj9A7hCfcFtXEqOuKqwMot9lrG4rBGSTaAVor/a02HXwwv
dbJPdujzqFtVKiLOyPm+rfbED66J1oh3tBokq58sUZJ1e5jNuzD0mZl6MRT22vus
jrsRHj6SSYswu5GIRgQYEQIABgUCPzqIGAAKCRBiIvi16frxKUidAKDKhtxUKnLW
Ts5KMegPmGYT+FPxCwCdHFq/iDHlss/oLiji9SZY0dXTPlk=
=R8LS
- - -----END PGP PUBLIC KEY BLOCK-----


* How You Can Help the GNU arch Project to Continue

https://www.paypal.com/xclick/business=lord%40emf.net&item_name=support+for+arch+and+other+free+software+efforts+by+tom+lord&no_note=1&tax=0&currency_code=USD


  The arch project is financially supported entirely by contributions
  from the general public.  Financial help, if you are able to provide,
  is both very much needed and very much appreciated.  You can find more
  information about this at:

            http://regexps.srparish.net/www
     or     http://arch.quackerhead.com/~lord


  The arch project is also, significantly, supported by contributions of
  labor by a substantial number of volunteers (listed in the ChangeLog
  and in the =THANKS file -- both of which, I fear, are doomed to be an
  incomplete record).  A community of contributors has helped arch to
  advance far more rapidly than would otherwise have been possible.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (FreeBSD)

iD8DBQE/8KkvYiL4ten68SkRApPHAJwIILSIK/URikZ0eCv3A/0MfxM+UwCZAc09
JvYrHH8yJj6qWx4wYmMviKk=
=nJUH
-----END PGP SIGNATURE-----