Re: Suggesting that feature/tree-sitter be merged (was Re: Tree-sitter and major mode inheritance)

[Yuan Fu]

> On Nov 18, 2022, at 1:54 PM, Jostein Kjønigsen <jostein@secure.kjonigsen.net> 
> wrote:
> 
> Hey everyone.
> 
> I know this has been said before, by people which by far has been 
> contributing much more than me... But I still don't think it's a good idea to 
> replace the implementation in existing major-modes with tree-sitter 
> implementations, nor selectively activate tree-sitter in major-modes prone to 
> inhetitence and derivation.
> 
> Me and Theodor faced these same issues with "our" C# and TypeScript 
> major-modes, and the only "clean" way we agreed we could make this work was 
> to create wholly new implementations. I can come up with many good, objective 
> reasons for this, but I think Theodor has already represented this view 
> fairly well.
> 
> While for the sake of brevity, I'll not diving deeply into this particular 
> thing, I will say this: A new tree-sitter based major-mode free of 
> compatibility concerns allowed us to create entirely new major-modes fixing 
> most of our existing bugs, faster than we before would be able to fix a 
> single bug. My personal view is that mixing existing major-modes with 
> tree-sitter represents absolutely the worst of all worlds. It maintains all 
> existing complexities, provides us with very few benefits, and at the same 
> time adds complexities we didn't use to have. To me, that's a net negative.
> 
> Somewhat surprising to me, I see this is a somewhat controversial point of 
> view and not as clear cut a matter as I would have expected it to be. I 
> realize and respect that final decisions in these matter might take some time 
> to mature. Time which given our current approach detracts from the momentum 
> tree-sitter has been having.

Hey Jostein,

Thank you very much for your thoughtful input! I originally thought of 
tree-sitter as merely a mean/tool that major modes could use to enhance their 
features. But in reality it seems that tree-sitter replaces a large enough 
chunk of a major-mode’s responsibility, which has caused all these 
compatibility problems we’ve encountered. Examples include undoing cc-mode’s 
setup when turning on tree-sitter, invalidating many existing major-mode 
variables, and the major mode inheritance problem I presented in the previous 
message. 

These difficulties has nudged us to decouple the tree-sitter and 
non-tree-sitter version further and further, from trying to make tree-sitter a 
feature that enables/disabled in a major mode by a command, to a major mode 
configuration that causes the major mode to setup differently, to my suggestion 
of using separate major mode but share a single virtual parent mode.

I think using separate major mode but share a single virtual parent mode gets 
the benefit of both worlds:
1. Tree-sitter and non-tree-sitter are separated in different modes, meaning it 
gets all the benefit Jostein and Theo described.
2. Backward-compatible. Existing configuration to the non-tree-sitter version 
works as they do before. (Here I’m mainly thinking about hooks: Hooks added to 
c-mode-hook still runs in c-native-mode.)
3. Solves the inheritance problem I described.
4. Minimal changes to the existing modes.

> At this point poor Yuan Fu here has spent quite a bit of time and effort 
> getting a core tree-sitter interface into Emacs. I would really hate to see 
> all that effort be for nothing, because we end up conflating a creating a 
> core tree-sitter feature with how this feature should best be employed in 
> subsequent major-modes.
> 
> So in the name of pragmatism, I propose a compromise of sorts.
> 
> Instead of waiting for "every" major-mode to be re-implemented into a 
> tree-sitter derivative in the feature/tree-sitter branch before we merge... 
> How about we just accept the current "core" tree-sitter implementation as 
> good enough, and consider merging that to git master as is.
> 
> This will allow us to land one important mile-stone, while giving us the 
> head-room to further discuss how we should best 
> implement/reimplement/"upgrade" existing major-modes to take advantage of 
> tree-sitter.
> 
> It will also allow third-party packages to make use of tree-sitter in Emacs 
> core instead coming up with its own implementation, beginning a defacto 
> standardization of this new API (which I may note already has a competing 
> implementation in MELPA).
> 
> That is to say: We should land the core library, with holding it hostage to 
> all its possible consumers being implemented.
> 
> How about it? Are there any good arguments for NOT merging 
> feature/tree-sitter at this point? :)

The good news is, feature/tree-sitter is merging in a few days! And we’ll make 
further improvements on the master. So rest assured! :-) I think we can at 
least get C, Python, Javascript, Typescript, Bash, JONS and CSS to go with the 
up coming release, largely thanks to Theo’s (and tree-sitter’s) productivity. I 
personally don’t know enough of C++ and Java to polish them, but they have a 
good chance too.

Taking a step back, I think developing major mode support with the tree-sitter 
API is a good idea. We found countless shortcomings of the API when developing 
major modes with it.

Yuan