Re: [Gnu-arch-users] arch with 'special files'

[Tom Lord]

   From: Josh England <address@hidden>

   No, I don't want backups...I really do want revision control.  Here's
   the scenario:  I've got software (http://onesis.org) that enables a
   single root filesystem to be used without any change on many
   functionally different machines (diskless clusters in my case).
   Consider the (very real) situation where there are many admins all
   making changes to this one root filesystem.  I'm want documented
   changes, the ability to incorporate only particular changesets, and the
   ability to roll back to any previous revision.  I want revision control.

That's a nice tidy summary.

   I've looked at several SCM systems (including Subversion and Bitkeeper),
   and I think Arch sounds very close to the solution I'm looking for. 

I agree.  Others have pointed out some areas where Arch contains code that
prevents it from directly supporting your application today.

The engineering decision(s) (under the umbrella hypothesis that using
Arch *somehow* is the best choice) might be choosing between

  (a) modifying Arch internals to iron out the details

  (b) building a thin layer of goo over core arch so that, without
      modifications, it is useful to you.

I am interested in (a) in the sense that, having read your use case,
I wonder if some of what you need that arch currently lacks should be
built in.   I can spend a little time chatting about the idea now;  
I think it unlikely I will have time to really dig in and write code for
it very soon (although things are always changing....).

/Note:/ mere discussion at this stage is not obviously pointless
because there is a forseeable point during the librification effort
(the stage at which we do improvement to the `inventory'
implementation) at which, if the (a) option looks good, it will make
sense to start coding for it in core arch.   (The librification effort
is basically a process of rewriting core arch, using the existing code
as a starting point, but cleaning it up an aweful lot.)

That said:

(b) may be a very practical option for you if your goal is to have something
working quickly.

It seems to me that for (b), you need only one tool (with two functions).
Let's call this tool `specialdeal'.

  /specialdeal/ functionality:

  (1) --freeze

      Given a filesystem which might contain special files,
      produce a text file which describes the location and 
      nature of those special files.   The format of this text
      file should be such that if files are produced for
      two different filesystems, it will be meaningful and 
      useful to `diff' those two files and to use that `diff'
      output as input to `diff'.

  (2) --thaw

      Given a file created by `--freeze', re-instantiate the 
      special files.


Approach (b) adds a layer of indirection which all of your sysadmins
are likely to need to know a little bit about, at least --- but it
also factors your solution out from arch development itself.
Conceivably (but don't count on it) a (b) soln might even be
"rev-ctl-system-independent": portable to svn, for example.


   The problem is I need to retain permissions on all files, and be able to
   archive special files (mainly device files).

John gave a good hint/starting-point about the issues.  For example,
what does it mean to `diff' two fifos or devices?  Suppose a given
device file changes its major or minor device number: should the
`diff' record that?  What should the `patch' process do?  (That kind
of thing.)

You're aware of those issues, afaict:

   You're right.  These kinds of files would need special handling, but it
   wouldn't be very difficult.  The code snippet you referred to could be
   extended to handle 'special' files as well.

Indeed.  It's necessary to have a rigorous idea of what `mkpatch'
and `dopatch' are supposed to do (our `diff'-like and `patch'-like
components).

   I'm not familiar with all the internals of arch, but all of this is
   conceivable and should be fairly straight-forward since tar is the
   file-store.  Tar is fully capable of storing device files and retaining
   permissions.  The 'special' file diff for a FIFO would just check
   existence.  A 'special' file diff on a char/block device would just look
   at the major/minor numbers of the device.  Sockets would still be
   ignored.

   Changesets of 'special' files would simply invoke 'mkfifo' or 'mknod' as
   appropriate instead of 'patch' (or whatever).  This is, of course, very
   UNIX-centric, but all the code could be #ifdef'd out for other
   platforms.

What if I diff two devices, which `arch' thinks are the "same file", 
but one is block and the other character?  What if I patch a tree with
`block' version and the patch says "add a char device here"?  Is that
a conflict?  Does the file silently change?

   > But really that would be easier to do by having a script that you run
   > which takes inventory of whatever you want, and then puts that into a
   > text file. Then you use arch to manage the text file.

   Certainly do-able, but not ideal.  Like I said, I want to stuff an
   entire distribution into an archive.  I don't just want to synchronize
   machines...rsync is suitable for that.  I want to allow any remote
   machine (administrator) to 'check out' a fully configured OS image, make
   any changes to it (that could be committed back to a main archive), and
   then boot their cluster into that image (that part I've got covered).
   They could make changesets for their own particular needs and they could
   commit any fixes/updates back into the main archive.

I don't see where what you want and the idea of using text files (aka my 
(b) solution, above) contradicts your usage scenario --- other than 
that it will likely introduce a couple of details that your admins will
have to be at least a little aware of.   (E.g., maybe you'd wind up with 
a rule like "after merging, run `specialdeal --thaw').


   (john speaking)
   > Arch also doesn't maintain owner/group. Again because that doesn't make
   > sense in standard revctl.

That's right but I'm not in principle opposed to owner/group support
if it is optional and clean.

   Revision control doesn't apply only to source code anymore.  I've found
   several instances of people doing exactly what I'm looking for by
   hacking all kinds of scripts around a CVS repository.  That sounds far
   to gruesome to me.  I'd like a clean, fully revision-controlled, linux
   root filesystem.  Conceptually, I think this doesn't sound too hard.  Of
   course the devil is in the details.  What do you think?

I think the worst case is you wind up with roughly the same thing
("hacking all kinds of scripts around" an Arch archive.)   

In better cases: I wouldn't be surprised if more exploration of your
use case results in feature ideas for Arch that would reduce the need
for such scripts.

The bottom line issues are: (1) getting a clear idea of what those
feature for core arch would be; (2) figuring out how to get them done
given everything else going on in arch development.

-t