Re: Proposal to turn off AOT in clojure-build-system

[Steve George]

Hi,

I don't think I'm making any progress convincing you, and I'm not enjoying the 
interaction so I'm going to take a few days off from this thread.

I've tried to ask the Clojure community for a definitive expression of what 
they think Linux distributions should do with byte-compiled libraries. We'll 
see if there's a response, this is a public forum so feel free to chime in and 
express clearly your perpective:

  
https://clojureverse.org/t/should-linux-distributions-ship-clojure-byte-compiled-aot-or-not/10595/1

Thanks,

Steve / Futurile

On 22 Feb, Maxime Devos wrote:
> >> [...] 
> >> > But, I would like to draw attention to this thread on Clojureverse as 
> >> > the best source I could find:
> >> >Alex Miller is the main community manager for Clojure, and is a 
> >> >maintainer of the core libraries, so his perspective is key. He notes 
> >> >that, AOT code is tied to *specific versions of Clojure*:
> >> >
> >> >  "AOT'ed code is that it is inherently the product of a particular 
> >> > version of tthe Clojure compiler ... I would recommend NOT AOT compiling 
> >> > libraries" [4]
> >> 
> >> This reasoning does not follow – yes, it is tied to the Clojure version, 
> >> so what? Guix automatically rebuilds dependents when the dependency (in 
> >> this case, the Clojure compiler) changes.
> (...)
> 
> >I think this preceding sentence is the heart of different assumptions 
> >between us.
> 
> >The Clojure packages we haave are for developers writing applications 
> >(libraries and tools). The ecosystem has very few end-user applications [0]. 
> >As a Clojure developer I can use the Clojure tools from Guix, and a few 
> >libraries. We (and all the other distributions) have a miniscule portion of 
> >the Clojure/Java library universe [1]. This leads to the following usage 
> >scenarios:
> 
> >1. A developer installs Clojure from Guix, and uses libraries from outside 
> >Guix.
> They can install the JVM/Clojure and some common tools (like clj-tools-cli). 
> They will use libraries from elsewhere, including their own. AOT compilation 
> is a problem because of the issue of mixed AOT and non-AOT.
> 
> >2. A developer installs a Clojure from outside Guix, uses libraries from 
> >inside Guix
> This will cause problems because the Guix Clojure libraries will have been 
> AOT'd by a different version of the compiler. It's also fairly common to 
> install/use parallel versions of Clojure/jvm due to different deployment 
> needs, this is likely to cause difficult to find bugs.
> 
> My answer to (1) and (2) is:
> 
> (a) About “install Clojure from outside Guix + use libraries inside Guix”:
> 
>  If these libraries are AOT:
> 
> Don’t do that, then. If you mix-and-match binaries (in this case, .class 
> files) different distributions, you are free to do so, but when (not if, 
> when) things break, you get to keep the pieces.
> 
> If these libraries are just the .clj files (not AOT) (which as I understand 
> it is the standard situation): doesn’t seem a problem to me (see point (b)).
> 
> Adding source from other places is one thing (probably ok), adding binaries 
> is another (probably not ok, especially if unstable ABIs (see Clojure 
> compiler) and mismatched versions (see hypotheses of 2.) are involved.
> 
> (b) What is this “the issue of mixed AOT and non-AOT”? Do you have a source 
> on this? Besides Clojure supposedly, I haven’t ever heard of such problems 
> for any language – for example, there is no such issue with Guile and AFAIK 
> not for Python. I haven’t heard of any such issues for the Common Lisp 
> implementations either (though I haven’t checked), so this doesn’t seem like 
> a “Clojure doesn’t do hygienic macros” issue.
> 
> (c) Guix isn’t forcing anyone to use AOT’d libraries. If people really want 
> to assist in murdering the climate (or its a situation where total cost of 
> non-AOT is lower than AOT), they can unfortunately (*) simply do so applying 
> a recursive transformation that adds #:aot? #false flags everywhere or 
> whatever the exact argument is (**).
> 
> Given that this transformation has some legitimate use cases, this 
> transformation could even be a pre-canned procedure + transformation included 
> in Guix itself
> 
> (*) ‘unfortunately’ only applies to the first case. For the case in 
> parentheses, replace by ‘fortunately’.
> (**) IIRC is wasn’t #:aot? but some other name, but that’s not really the 
> point here.
> 
> (d) If a deployment need multiple versions of Clojure, then just fix your 
> deployment to make everything work with the latest version of Clojure. Or, if 
> you for some reason don’t do that, just use a (recursive) transformation to 
> change the version of the Clojure compiler used to match the Clojure that 
> will be used in the particular deployment.
> 
> (e) You can simply add missing packages to Guix as the need arises.
> 
> >I can see the sense of compiling to byte code if it's an end-user 
> >application. In that case we'd want to make the start-up as fast as 
> >possible. Your comments seem to have this use-case in mind.
> 
> >But, today there aren't any such end-user Clojure applications in Guix.
> 
> That’s a shame. Hopefully that will change some day.
> Also, this is false, “clojure-tools” exists – developers are people too (I 
> don’t care about the “_end_-user” distinction – surely developers want fast 
> start-up as well).
> 
> Also, I _don’t_ have that use case in mind – I have efficiency in general in 
> mind, efficiency of the start-up is only a part of that.
> 
> The point is: most likely you will want the application to have AOT code, and 
> that library has dependencies. Furthermore, likely many of these dependencies 
> are dependencies of other applications as well.
> 
> Instead of redoing the compilation of N shared dependencies for each M 
> applications  (total work: ~N*M), by not participating in these Clojure 
> conventions, the total work of compiling dependencies can be reduced to ~N).
> 
> Let’s avoid joining the madness that is Go’s build system and Cargo(*).
> 
> (*) Rust is fine for compilation, it’s cargo that is the problem (at least, 
> Cargo as used by Guix) – antioxidant 
> (https://notabug.org/maximed/cargoless-rust-experiments) avoids the sheer 
> inefficiency of Cargo.
> 
> >> >I believe this means that with AOT code on, any user who installs a 
> >> >different version of Clojure from the one that we used to AOT the 
> >> >libraries *may* have problems.
> >> 
> >> Unlike, say, Maven, this situation simply does not happen in Guix, because 
> >> we don’t just download binaries and call it a day (except for some 
> >> bootstrapping stuff, but that’s not relevant for Clojure AOT), because we 
> >> have functioning recompilation of dependents, because of shebang patching, 
> >> because binaries that are to be invoked should not rely on the ambient 
> >> CLASSPATH / LD_LIBRARY_PATH / etc. and, if, the underlying binaries do 
> >> rely on that, they are wrapped (see wrap-program) to set them (or, at 
> >> least, they should be, you might find some bugs in this department if you 
> >> go looking).
> >> 
> >> Even if they aren’t wrapped, then in that case the dependencies are 
> >> propagated-inputs, and you can only have a single version of a propagated 
> >> package in the same environment (barring stacking environment shenanigans, 
> >> but then you are looking for it and/or you can just report a bug about how 
> >> the binaries should be wrapped/classpath should be patched in/...).
> > 
> 
> >In this paragraph you're assumption is that a Guix user is only using 
> >libraries from within Guix. Hopefully, I've shown why this assumption is 
> >unlikely above.
> 
> You haven’t. You have shown that other situations exist, not that this 
> situation is unlikely. To demonstrate likelihood / unlikelyhood, you need 
> some statistics (or a very thorough argument on why people wouldn’t just make 
> a package for missing dependencies – it’s very simple, especially if you 
> aren’t sending them for inclusion.)
> 
> >You also mentioned Debian, and @e.hashiman [2] said that Clojure libraries 
> >are not AOT'd on Debian, while applications are. From what I can find there 
> >are 130 packages in Debian with the word Clojure in them [3]. I looked at a 
> >selection and it seems true that Debian does not AOT libraries (and I can't 
> >find any Clojure 'apps'). For completeness I also checked what Clojars, the 
> >main distribution archive for Clojure developers, does:
> > [...]
> 
> OK ... and? Other people doing things differently doesn’t mean those 
> different approaches are better. If Debian mentions somewhere the reason 
> _why_ it doesn’t AOT libraries, that reason can be investigated and perhaps 
> carried over to Guix, but it’s the ‘why’ that’s important, not the ‘it 
> doesn’t’.
> 
> > Does this help clarify why I'm asking to change the default?
> 
> Yes. It appears you are unfamiliar with the “transformations” functionality 
> of Guix, which allows for easy rewriting of packages to whatever Clojure 
> version you want, eliminating compatibility concerns or even turning off AOT 
> completely.
> 
> (This requires the relevant versions of Clojure to be packaged in Guix, but 
> that shouldn’t be too much of a problem.)
> 
> It also appears you are unfamiliar with Guix importers, which easily make 
> package definitions (somewhat low quality w.r.t. descriptions, with lack of 
> verification of freeness, etc., but usually otherwise fully functional). I 
> don’t know if an importer for Clojure exists but it shouldn’t be too 
> difficult I think.
> 
> Best regards,
> Maxime Devos.