Re: Tree-sitter integration in python.el

[Yuan Fu]

> On Sep 26, 2022, at 12:10 PM, Jostein Kjønigsen 
> <jostein@secure.kjonigsen.net> wrote:
> 
> On 22.09.2022 20:42, Yuan Fu wrote:
>> Hi, 
>> 
>> I’ve added tree-sitter version for font-lock and which-func in python.el. 
>> And I’d love to hear some feedback from python.el maintainers. Specifically, 
>> does it look right and which other part of python.el could actually benefit 
>> from a parse tree? I wrote a tree-sitter imenu indexer for python and it 
>> performed worse than the current one, presumably because it traverses the 
>> whole parse tree whereas the current one only scans the buffer once or so 
>> and do some regex matching.
>> 
>> Here is the commit: 
>> https://git.savannah.gnu.org/cgit/emacs.git/commit/?h=feature/tree-sitter&id=1cdb24fe35a9ff2e4f92c5acc93a5a5b0e70d93f
>> 
>> 
>> Yuan
>> 
>> 
>> 
> Hey Yuan.
> 
> Thanks for putting in all this work into tree-sitter in Emacs!
> 
> Trying the latest Emacs-version in feature/tree-sitter I got an error I tend 
> to get "a lot" with tree-sitter based modes, which I hoped bundling things 
> with Emacs would solve, namely obtaining the original shared-object 
> containing the compiled grammer.
> 
> Like for your python-mode, I get this:
> 
> File mode specification error: (treesit-load-language-error python 
> (/home/jostein/.emacs.d/tree-sitter/libtree-sitter-python: cannot open shared 
> object file: No such file or directory 
> /home/jostein/.emacs.d/tree-sitter/libtree-sitter-python.so: cannot open 
> shared object file: No such file or directory libtree-sitter-python: cannot 
> open shared object file: No such file or directory libtree-sitter-python.so: 
> cannot open shared object file: No such file or directory))
> 
> I realize third-party modes are on their own, but when tree-sitter is 
> compiled with Emacs, I would at least expect the Emacs-build to also produce 
> these .so-files.
> 
> What are your thoughts on how we can best, across the Emacs-verse, provide 
> these libraries? Or at least for the modes which are bundled with Emacs 
> itself?
> 
> Being a tree-sitter based-developer myself, I know where I can go to get this 
> compiled to make the mode runnable, but surely that's not how we can deploy 
> this en-masse. Most people will be stuck at this point, and we will need to 
> come up with a better answer.
> 
> -- 
>  Kind regards
> Jostein Kjønigsen
> 
> jostein@kjonigsen.net 🍵 jostein@gmail.com
> https://jostein.kjønigsen.no

This has been discussed before and our conclusion is to treat language 
definitions like other dynamic libraries, expecting package manager to install 
them. There are a couple of reasons. Tree-sitter library and language 
definitions must be on the same “protocol” version to work. I expect this 
version to change slowly, but in principle we don’t want to bundle a language 
definition that could conflict with the tree-sitter library provided by the 
system. Also, as a dynamic object file, it is machine-dependent. I don’t know 
if we bundle anything machine-dependent in Emacs distribution, but at any rate 
it is unusual. 

I do have some ideas to make it easier for users, like hosting them on ELPA or 
NONGNU ELPA, and user can install them by package-install. Eg, a package 
tree-sitter-installer-linux, and M-x tree-sitter-installer-linux RET c RET will 
install C language definition for you. Stefan, WDYT?

Yuan