Re: [Chicken-hackers] new CHICKEN 5 egg installation

[Mario Domenech Goulart]

Hello Felix,

On Wed, 22 Feb 2017 15:20:11 +0100 address@hidden wrote:

> I have now a first prelininary version of the new egg installation
> tools for CHICKEN 5 working, and before we go on, I'd like people to
> review and reflect on the changes that have been made so far. I also
> need some advice regarding a number of issues...
>
> The branch is "chicken-5-new-egg-install"[1], a preliminary selection
> of eggs that have been ported to new the description file format are
> available at [2] (thanks to Mario + Kooda, who have provided a bunch
> of eggs). Note that this repository is not final, yet. Eggs may be 
> removed or deprecated or will need considerable changes, should
> more disruptive changes be made in CHICKEN 5 (let's hope not...)
>
> Basic uses of chicken-install/chicken-status/chicken-uninstall should
> work. I have tried to document everything [3], please tell me if 
> important information is missing or something does not match the
> current implementation.

Thanks a lot.

> Issues:
>
> * Source-level depedencies inside eggs can currently not be specified,
>   but component-level dependencies can, this might be enough.
>
> * When a test in a depency fails, the installation process still continues,
>   should the whole operation abort?

Is this related to the question ""-test" implies that all tests are
executed ..."?  I think the current behavior in C4 is nice: tests are
only executed for the eggs given on the command line.  I think in C4 you
actually have to install eggs before testing them, no?

> * What should be the default optimizations, or should there be a
>   default optimization level at all? (this can be given in the .egg file);
>   the same applies to compiled import libs, where the user can not
>   override the options.

I think the default should be csc's default, and users should be able to
specify the optimization level they want.

> * Is an option for not fetching/installing depedencies desirable?
>   This has been added recently to C4, what is the inention of this,
>   and how does it work if the dependency is required for building?

That option was added to C4 to provide a way for packagers to determine
what files belong to each egg.  That's not trivial to determine in C4.
With -no-install-deps, we can install dependencies in the canonical egg
repo and specific eggs in another directory (their dependencies are
loaded from the canonical repo).  With this, packaging an egg means
packaging the directory where the egg was installed.

> * "chicken-install -init" is currently not implemented. Do we need this?

Salmonella uses this, but see my comments on the question about -prefix.

> * The following options are not implemented:
>     -keep-installed
>     -reinstall
>     -proxy  -username   -password   (use env vars instead)
>     -init
>     -deploy
>     -keep-going
>     -scan
>     -csi
>     -show-depends
>     -show-foreign-depends
>     -prefix

Ok.

> * "-test" implies that all tests are executed, including dependencies;
>    or should only those tests run that belong to the eggs explicitly given
>    on the command line?

I personally like the current behavior in C4 (run tests only for eggs
explicitly given on the command line).

> * "extension-information" actually retrieves information for a whole egg.
>   Should this be renamed, or should it extract information for a particular
>   component?

If I understand correctly, "extensions" are library components of eggs.
Is my understanding correct?  In this case, I suppose we should use
egg-information to obtain information for a whole egg, and
extension-information for the library component (if it makes sense).

> * I would REALLY REALLY like to get rid of "-prefix", "chicken-prefix", 
>   CHICKEN_PREFIX, etc. as they cause pain and confusion, but
>   Mario says that it is important for Salmonella.

Currently, salmonella uses -init to initialize a private local egg repo
for tests, and uses -prefix to install eggs in it.  In the non-cached
mode, it sets that private egg repo empty for each egg it tests.  That's
how salmonella tests eggs in a fresh environment, and that's how missing
dependencies are caught.

I don't know another way to implement this feature.  Things like chroots
and a special CHICKEN installation for salmonella have been suggested,
but they are not really good options in my opinion.  chroot is not
portable and would require administrator privileges.  A dedicated
CHICKEN installation for salmonella would totally kill the convenience
of using salmonella as an egg checker for casual users.

Another suggestion was about making chicken-install generate a list of
modules imported by egg components.  With this, it would be possible to
cross-check the generated list of module dependencies against the
dependencies specified by `dependencies' in .egg.  The list of imported
modules is actually generated by csc.  So, to test `test-dependencies',
chicken-install would have to make tests/run.scm go through csc.  And if
we have custom build scripts in CHICKEN specified via `custom-build'
(e.g., scripts that use the make egg, setup-helper etc.), how can we
detect dependency problems?  I don't know how many times I saw eggs
breaking because their .setup used some egg that was not specified in
`depends'.

So, I don't really know how to go about salmonella.  I'm completely open
to suggestions.

> * It has been suggested to sign eggs (or egg hashes). I don't know enough
>   about this to decide how to proceeed. If such a feature is desired, it 
> should
>   be added early (standalone crypto code seems to be available, e.g.
>   "tweetnacl")

That'd be nice.  Are you also considering tarballs?

> * Testing on Windows + OS X still has to be done, especially the build/install
>   script generation for Windows/cmd.exe has never been tested. "chicken-do"
>   needs to be ported to Windows, as well.
>
> * The old system allowed for so called "syntax-only" eggs, extensions that
>   only provide syntax (and thus don't require runtime code to be loaded),
>   like miscmacros and matchable. This is currently not implemented, and
>   I'm not sure whether the extra costs are worth the trouble.

Some assorted comments:

* Do we need `doc-from-wiki'?

* chicken-uninstall should uninstall _eggs_, no?

  I ask because the documentation for chicken-uninstall and chicken-status
  mention "extension":

    -exact
        match extension-name exactly (do not match as pattern)

  By the way, I think the default behavior should be exact matching
  (http://bugs.call-cc.org/ticket/1093).

> [1] 
> https://code.call-cc.org/cgi-bin/gitweb.cgi?p=chicken-core.git;a=shortlog;h=refs/heads/chicken-5-new-egg-install
> [2] https://code.call-cc.org/cgi-bin/gitweb.cgi?p=chicken-5-eggs.git;a=summary
> [3] 
> https://code.call-cc.org/cgi-bin/gitweb.cgi?p=chicken-core.git;a=blob;f=manual/Extensions;h=38256f9b7a9370d38e63ee5062aaa7b27fa307ba;hb=refs/heads/chicken-5-new-egg-install

PS.: I'm having trouble to compile the chicken-5-new-egg-install branch
using 4.12.0 to bootstrap chicken (I can compile the chicken-5 branch,
though).  Is this a known issue?  It's possible that I'm doing something
wrong, naturally.  I tried to rebase against chicken-5, but git scared
me a little.  Judging by b35822385, I suppose git has not been treating
you well. :-)

All the best.
Mario
-- 
http://parenteses.org/mario