[Gnu-arch-users] Re: in-tree pristines fatally wounded (merge-fest etc)

[Pau Aliagas]

On Tue, 2 Dec 2003, Tom Lord wrote:

> A good goal, I think, might be this:
> 
> * goal
> 
>   Make it trivial for a new user to create an environment in which:
> 
>       ~ none of their project trees ever contain pristine copies
>         ~ they have a library which operates sparsely
>         ~ --hard-links works for them and its easy to extend their
>           default environment to account for working on multiple 
>           devices
>         ~ they don't have to write or install any hook functions
>         ~ if their library or libraries start getting too large,
>           they can issue a single easy command to prune them back down
>         ~ it's trivial to make hardlink project trees their default
>         ~ all of this is done upwards compatibly

I fully agree with all of them. After we have all this in place we can be 
confident on managing big projects ;)

I would be very flexible with the compatibility. People may have to
recreate the libraries to include the inode signatures or just to be able
to use these new features. It's not a big loss or effort to recreate the
libraries. Pristine tree will have to be deleted too and I'm sure that
most of people ignore what they are and that they are in their working
trees. So we could five a few hints on how to do that for the existing
user base.

>   I think we can do that:
>
> * library paths
> 
>   Yes, I do plan to either merge/extend or simply rewrite library path
>   support.
>
>   In fact, as I've tried to write up these ideas for this message, it
>   occurs to me that we can actually do without the hooks quite nicely!
>   (I'll still keep the hooks -- it's just that I think with a
>   sufficienly good path mechanism, the default behavior can eliminate
>   99% of the need to ever actually use the hooks!)
> 
>   I think we, in fact, want _two_ library paths:  one for search-only
>   and one for add-precedence.
> 
>   The total path when just searching for a revision is the
>   concatenation of the search-only path with the add-precedence path
>   (in that order).
> 
>   The total path when automatically adding a revision is the
>   concatenation of the add-precedence path with the search-only
>   path (in that order).
> 
>   (Many users will be happy with just one of those two paths but
>   in some situations it is useful to say, in effect, "i prefer
>   to add to low-priority libraries but take from high-priority 
>   libraries.")
> 
>   When searching, if searching for --hard-links purposes, return
>   the first revision found that is on the appropriate device;  
>   otherwise, just return the first revision found.
> 
>   When auto-adding, add to the first of the following libraries
>   found (on the appropriate device if for --hard-links purposes):
> 
>   ~ the first library on the path that contains the revision's 
>     version
> 
>   ~ the first library on the path that contains the revision's
>     branch
> 
>   ~ the first library on the path that contains the revision's
>     category
> 
>   ~ the first library on the path that contains the revision's
>     archive
> 
>   ~ the first library on the path
> 
>   I also think we therefor need some new commands:
> 
>       tla library-add-archive
>       tla library-add-category
>       tla library-add-branch
>       tla library-add-version
> 
>   (and the symmetric `delete' commands).

I think that this approach, although doable, is complicated. 
Implementation complications would be undoubtably, overcome.
But I think it's not worth adding these bunch of commands for something 
that can be achieved with a simple hook.

What I propose (you can even see the implementation) is a two way
communication mechanism in hooks. Now hooks cannot return anything to the
caller; if you enable the communication from the hook to the caller, the
hook can return, for instance, the path to the library. I did it using
file descriptor 3, but it could be any other mechanism.

My reasoning is that it is very easy to do and to understand and very
flexible. Whenever you want to fine tune the cached revisions locations, 
doing it in a hook script, with full control, will be much easier than 
mastering a path jigsaw.

>   And how about a per-library or per-library-part flag that says
>   whether --sparse is the default, along with some other parameters:
> 
>       tla library-policy [--every <N> | --decay] \
>                            [--greedy] \
>                            [--sparse=yes|no] \
>                            <library>  \
>                            [<archive>|<fqcat>|<fqbranch>|...]
> 
>       tla library-lock [--unlock] <revision> [<library>]
> 
>         tla library-prune [<library> [<archive>|<fqcat>|...]]
> 
>   where library-prune attempts to reduce the number of 
>   revisions in (optionally: some part of) a library following
>   the rules:
> 
>         ~ keep all locked revisions
>         ~ keep all revisions whose link count indicates that
>           there are hardlinks from other library versions or
>           from project trees
>       ~ within those constraints, get as close as possible
>           to (depending on the policy) every Nth revision
>           or an exponential decay from most recent to 
>           oldest

This would be, obviously, a plus, something nice to have. But is sounds 
like a lot of work to code in arch *-)

But if you take the hook approach..... tadda!!, you can do it: policy is
in your script, no need to understand and configure all these options. 
Nice, isn't it?

Policy belongs to the user in both cases, but IMHO it's easier to 
understand that you can do this in a script than a pile of options.

Even something as simple as a maximum revisions per version would do it.
The question is forgetting about maintenance, so maybe we could start with
this simple proposal and then, if required, go further. Let's not do 
overengineering, maybe too few people really need this setup.

I'd go for the script, listen to people's feedback and then decide.

>   With those low-level mechanisms in place, we can imagine a 
>   high-level convenience command.   Something like:
> 
>       tla project-mirror [params] <name> <project-spec>
>   
>   where the <project-spec> is a list of archives and/or
>   fully qualified categories, branches, or versions and
>   the params specify:
> 
>       ~ if and where to create local archive mirrors
>         ~ if and where to add the archives, categories, etc.
>           to particular libraries
>         ~ what library caching policies (if any) to apply 
>           to the archives, categories, etc.
> 
>   Perhaps then:
> 
>       tla sync-project <name>
> 
>   to issues all the necessary push-mirror commands and
> 
>       tla prune-project <name>
> 
>   to issue appropriate library-prune commands.

Mmmm... maybe these should be external scripts if they don't add anything 
new. Just let's distribute a scripts directory with all these convenient 
utils, but let's not complicate arch with thes aggregation commands. Isn't 
that what itla is all about?

>   Finally, about the --greedy option to library-policy: if some part
>   of a library is greedy, that means that any demand at all for a
>   pristine copy of some revision that can go in that library or
>   library-part is satisfied by auto-adding to the library.  For
>   example, if you have a greedy library, `what-changed' won't
>   every build and then delete a "scratch copy" of some revision -- 
>   it'll just auto-add it to the greedy library.
> 
>   So a default "new user" set-up can create a single library,
>   make is sparse if that's the consensus, set it up for 
>   exponential-decay caching-policy, and make it greedy.
> 
>   The net effect is that such new users will never have an
>   in-tree pristine and `tla library-prune' will have a good
>   default behavior for them.

That can be easyly achievable if we distribute arch with nice defaults. 
For instance with my-revision-library set by default to ~/.arch-cache and 
a hook script that adds libraries when required.

> * make --hard-links the default
> 
>   I don't think that that should be the default default but it should
>   be easy for a user to make that a personal default:
> 
>       tla want-hard-links [never | sometimes | always]
> 
>   Never means nevery make a hard-link tree except if --hard-links is
>   explicitly provided.
> 
>   Sometimes means to make hard-links if there is a handy library 
>   revision on the same device, but otherwise just copy.
> 
>   Always means to always make hard-links.  If there is no library
>   on the appropriate device, issue an error message advising the user
>   to create one (and suggesting how).

I agree with this one. Setting it as default is dangerous, so adding this 
command looks like a good way to do it. As my vision of library paths is 
different from yours, tla want-hard-links [yes | no] wold work for me; the 
rest would be responsablity of the hook script.

> * making --sparse the default
> 
>   Nah.  That's a coin-toss issue.  Neither sparse or non-sparse
>   is the right default.   But I'm willing (see above) to make
>   sparse the default for a particular library or part of a library.
> 
>   (Other than that, though, non-sparse wins as the default because
>   it is upwards compatible.)

I think that people keep on going forward, so adding all the old revision 
the first time seems exagerated. I would not expect it. So I would go for 
sparse by default.

Upwards compatibility? What do you mean? I don't see loss of 
compatibility.

> * how many library hooks?
> 
>   I don't care whether it is one or three but the hooks should
>   take additional parameters which we can work out as we go along.

I only see one hook, as explained in the previous email, but these are
trivial details, as you say.

> * "returning" a path from the library hooks?
> 
>   That just strikes me as awkward.   I think the library-path 
>   search algorithm described above is simpler.

It seems awkward, but once you start using it you realise of the opposite. 
Try it and feel the power of simplicity. It's so easy and at the same time 
so powerful! Gosh, I'm marketing myself :)

>     > I completely agree that now we are ready to use these huge trees and be 
>     > superior to other non-free alternatives: less space and more speed, 
> what 
>     > else can you ask for?
> 
> :-)

Maybe you should contact Sourceforge to help them to migrate from their
CVS hell to the arch heaven. That would be a real proof to some agnostics. 
And an itch to others ;)

Pau