Re: [elpa] main 8f4cb59: * elpa-packages (counsel, ivy, swiper): Auto-sync.

[Stefan Monnier]

>>     * elpa-packages (counsel, ivy, swiper): Auto-sync.
> Sorry, I realised only after the fact that the corresponding externals
> in elpa.git had significant local changes in the form of deleted files,
> diverging Version headers, etc., so I'm guessing :auto-sync won't work
> for these packages.

Indeed.  When I saw that commit this morning I thought (naively?)
that maybe you had sorted it out :-(

> Should I revert this commit in the meantime?

I think so, yes (it's not terribly harmful because the sync will just
fail with "can't fast forward", but it's not a very valuable use of
resources ;-)

> It gets even more complicated: the existing ivy-hydra.el and the new
> ivy-avy.el in swiper.git were apparently designed as optional separate
> packages, so I've now created the corresponding external branches in
> elpa.git (but not yet added them to elpa-packages).
>
> What is the best way to handle this complexity (that of having multiple
> packages provided by a single upstream repository) in elpa.git?
>
> For example, should swiper.git have separate branches for each package's
> subtree, so that elpa.git can (hopefully) just fetch and fast-forward to
> them automatically?
>
> Should things continue as they are now, with elpa.git deleting files
> locally and swiper.git blissfully unaware of the fact?

Good question.  The scripts used for GNU ELPA were designed with
a dual-use in mind, where OT1H they can take the elpa.git and generate
the ELPA tarballs and OTOH they can take an elpa.git clone and build the
packages for use in-place.

The "use in place" feature is not used very commonly, but it is the most
natural use case for the developer (which reduces the difference between
what the developer uses and what the users use, thus reducing the risk
for errors introduced by the packaging) so I find it important.

To me the two best options are:

- Split the upstream repository so we get one repository per ELPA package.

- Get rid of the split between the ELPA packages (so you again get one
  ELPA package per repository).

Another option is to have separate manually-synced branches in the
upstream, one per ELPA package.  This is basically the same as what we
have currently, except that the manual syncing is done between the "main"
branch" and the "for-elpa branches" (all within the upstream repository)
rather than between the main branch in the upstream and the elpa
branches in `elpa.git`.  To me, it seems to bring no benefit, but
I guess depending on your workflow (and access rights) it could make
a difference.

There's another option, which I dislike and introduces inefficiencies,
which is to make all those packages have the exact same content (so they
all have the same upstream branch and we can fast-forward it) and then
rely on the `:ignored-files` parameter in `elpa-packages` to filter out the
files we don't want in the tarballs.
Downsides:
- A full elpa.git checkout (like the one elpa.gnu.org keeps to build the 
tarballs, or the
  one I keep on my machines to "install in place" all the GNU ELPA packages)
  would contain duplicate copies.
- The `:ignored-files` doesn't have any notion of "negation" so you
  can't say "ignore all but counsel.el", making those lists of
  ignored files annoying to maintain.
- For the "install in place" case, those duplicate copies get
  redundantly byte-compiled as well.
- For the "install in place" case, the "ivy.el" file loaded by Emacs
  may not be the one I think: instead of `packages/ivy/ivy.el` the
  `load-path` may direct Emacs to choose `packages/counsel/ivy.el`
  instead (so I'd have to be careful to ask Emacs which file it's using
  before I start hacking on it, lest I'd start modifying one file which
  turns out not be used, which could lead me to tearing out my hair for
  a while).

> Or maybe adding something like MELPA's ability to specify which exact
> files to consider from the upstream repository?

That could be added, and could reduce the burden of managing the
`:ignored-filed`, yes.

It'd likely have to be done in this part of the elpa-admin.el code:

        (apply #'elpaa--call
               nil "tar"
               `("--exclude-vcs"
                 ,@(cond
                    (ignores
                     (mapcar (lambda (i) (format "--exclude=packages/%s/%s" 
pkgname i))
                             ignores))
                    ((file-readable-p elpaignore) `("-X" ,elpaignore)))
                 ,@(mapcar (lambda (r) (elpaa--make-tar-transform pkgname r))
                           renames)
                 "--transform"
                 ,(format "s|^packages/%s|%s-%s|" pkgname pkgname vers)
                 "-chf" ,tarball
                 ,(concat "packages/" pkgname)))

right at the end here, you'd probably want to list each part
individually instead of telling tar to include the whole subdirectory.


        Stefan