Re: Multiple versions of Org in load-path problem

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>> "Michel Schinz" <michel@schi.nz> writes:
>>>
>>>> Just for the record, I also ran into problems when installing Org 9.6
>>>> using Emacs' package system on top of an older version that came with
>>>> Emacs. If I tried to install it as usual (M-x list-packages, then
>>>> install the package from there), I had errors during compilation related
>>>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>>>> fatal error in an unrelated package.
>>>>
>>>> I managed to solve that problem by:
>>>> 1. uninstalling Org 9.6 and exiting Emacs,
>>>> 2. starting Emacs with -q,
>>>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>>>> 4. restarting Emacs.
>>>
>>> Interesting!  I tried this (essentially) and it worked for my case.  In
>>> my case, I had a built-in Org-9.3 and I was trying to use list-packages
>>> to install Org-9.6. I checked that using -q still added Org-9.3 to
>>> the
>>> load-path, but, since Org wasn't loaded, the install via list-packages
>>> worked.
>>>
>>> The question is what's the proper way of doing this without '-q'?
>
> [...]
>
>> I don't think there is any safe way to install an updated version of
>> org-mode other than
>>
>> 1. Use the -q approach outlined above
>
> Thinking about it, this only works if Org is in elpa as melpa (etc.) are
> not added to package-archives.  You'd have to do some handwritten elisp
> out of *scratch* to setup package-archives if Org-9.6 was still coming
> out of melpa. That's why this can only be labeled as a hack and not a
> solution.
>

Well, yes, if your going to use this technique to load a package which
is not in the default package archives you would need to add that
archive first. People who use this technique often just have a
'update.el' file which they load/evaluate when starting Emacs with -q. 


>> 2. Craft your init.el file such that org functionality is only loaded
>> when explicitly requested and always update as the first action after
>> starting emacs.
>
> In this case, something happened in package-install when trying to
> install Org-9.6 with a built-in Org-9.3.  During the compilation check
> (.el -> .elc) many files failed because the new 'org-assert-version'
> macro was not defined.  Sort of like, after package-install started
> working on Org-9.6, org-macs.el (where org-assert-version should be) got
> loaded *before* the new load-path had been set causing it to load the
> old one from 9.3.  Thereafter, everything went awry. 
>

You must have some custom code in your init.el or are using something
like use-package as package.el doesn't try to install or upgrade
packages during init by default.

Where people can come undone is when they are using use-package and set
the :ensure t option. In this case, use-package will not know about the
bundled version and will attempt to install org from ELPA. If use
package runs after org has already been loaded (possibly because some
other package has been loaded which depends on/requires org mode and has
loaded the bundled version) then things will break because you will end
up with a mixed version install. This is why I always ensure org is the very
first use-package in my init.el and it comes before any other code which
loads or initialises anything.  

>> The first approach is actually the easiest. The second is hard to get
>> right and very fragile because packages like use-package and more
>> specifically, other packages with leverage off org functionality, make it
>> impossible to reliably know exactly when org is loaded.
>
> Using ':after" in use-package is supposed to help that, but I'm not sure
> it is reliable.  Packages are often incomplete about what other packages
> it depends on.
>

You cannot rely on :after. The problem is, other packages may require
org functionality and will load org when they are loaded. This can be
very subtle as there are a lot of packages which only make very small
use of org mode, but even that requires that org mode is loaded. 

>> An approach used by many 'canned' distributions is to postpone package
>> updates. You have a function you run to check for updates which
>> generates a list of packages to update and writes that list to a
>> file. Each time emacs is started, it looks for this update list and if
>> it finds it, it installs packages updates at the very beginning of the
>> init process (before any of your other init.el code or custom
>> blocks). The process also looks for org in the list of packages to
>> update and if it is found, updates it first. 
>
> Probably doesn't work in this case as you would need to be able to use
> package.el suggesting that the load-path has been updated for all
> built-ins already.  The thing I note is that the load-path has already
> been updated for built-ins at the beginning of
> '~/.emacs.d/early-init.el', but the libraries haven't been loaded yet
> (unless needed). That's okay if the newer version of a package then
> cleanly replaces all the files in the old version.
>

It does work. Distributions like Sapcemacs use this approach and do not
have the mixed versions issue. I've not looked closely at their
implementation, but they well might do all this work as part of the
early-init.el file.


>> I don't think there is a safe way to load org mode after the init
>> process i.e. after booting emacs by M-x package-update.
>
> Where is package-update called in the boot process of emacs?  I don't
> see package-update in Emacs v2.7.
>

Sorry, I wasn't clear.

What I meant is that there is no clean and reliable way of running the
package update process after emacs has initialised. The package.el API
has changed, but previously, you would run M-x package-update-packages
to update installed packages. Now it is M-x package-update-all. With
many packages, you can run this command and update your installed
packages. However, more complex packages, like org-mode, cannot handle
this due to potential mixed version problems - a problem which has
existed for many years. 

By default, there is no automated package update setting (unless that
has been added in recent versions). There are some add-on packages which
can do this and the elisp to make this happen is trivial (but much
harder to do it and make it robust). 


>> I've had good success using straight.el. I had to be careful regarding
>> how I structured my init.el file (ensuring any straight stuff happens
>> first and the first use package stanza is for org. The main reason
>> straight works well for me is that my work flow is to do a M-x
>> straight-pull-all when I want to update my packages. This does a git
>> pull for all the sources, but does not do any build/install. This occurs
>> when I next start Emacs and because I have all the straight stuff at the
>> start and because org mode is the first straight-use-package, the update
>> and install happens before any other org functionality is loaded,
>> avoiding mixed version issues.
>
> Where do you get straight.el?  I don't see it in [m]elpa.

No, to use straight.el, you add some code to your init.el which will
retrieve the current straight.el file from the github
repository. Straight.el does not use the ELPA/NONGNU ELPA/MELPA archives
- at least not directly. It does use information from those archives to
build recipes which it sues to obtain the code. Essentially, straight.el
clones the package source repositories and builds it locally. It give a
lot more control and is really great if you want to modify the code
(basically, it creates a local fork). Of course, with all that extra
power, there is also additional responsibility placed on the user. You
may need to tell it which versions you want to use, which ones to 'pin'
to, which branches or fix merge conflicts when you have made changes and
want to update from the upstream soruces etc. I've been using it for
about 12 months and find it really good. YMMV.