bug#50459: 28.0.50; Python shell completion is incompatible with flex, orderless, etc.

[João Távora]

Augusto Stoffel <arstoffel@gmail.com> writes:

> On Fri, 10 Sep 2021 at 13:37, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
>> Augusto Stoffel <arstoffel@gmail.com> writes:
>>
>>> To alleviate this, the completion-at-point function could implement some
>>> sort of caching.  The difficult question is when to invalidate the
>>> cache.  I've attached one possiblility as a draft patch.  If the
>>> approach seems reasonable, then I'll format it properly.
>>
>> Would it be possible to do caching at a lower level instead of in
>> python-mode?
>
> I'm not an expert in this either, but I think the caching mechanism
> would be pretty particular to the circumstances of each completion
> table, so it indeed belongs here.  Maybe João can say more?

I'm sorry, I'm quite overloaded lately and can't read this long thread.
I've been participating in a few discussions about this and all I can
add here are generic comments, like the intepretation that I make from
the docstring of completion-at-point-functions

    Special hook to find the completion table for the entity at point.
    Each function on this hook is called in turn without any argument and
    should return either nil, meaning it is not applicable at point,
    or a function of no arguments to perform completion (discouraged),
    or a list of the form (START END COLLECTION . PROPS), where:
     START and END delimit the entity to complete and should include point,
                           ^^^^^^^^^^^^^^^^^^^^^^     
     COLLECTION is the completion table to use to complete the entity, and
                                                  ^^^^^^^^^^^^^^^^^^^
     PROPS is a property list for additional information.

As I've underlined, it's _that_ entity, not some other entity that the
"backend" aka "capf" should complete.  So:

a) while the entity is "the same" (START and END are the same and POINT
   is somewhere in between) then the backend can do caching inside
   COLLECTION (Eglot does caching between calls to try-completion,
   all-completions, and others, for example).

b) if the entity changes because the buffer has been changed, by any
   means -- including the means of an completion UI -- then the
   completion UI should re-invoke the capf to get a the new entity to
   complete and the new COLLECTION to complete it.  Emacs's default
   completion UIs do this, and so does Company, if I'm not mistaken.  If
   the backend is super smart and can be faster about responding to this
   new capf invocation validly given the previous respose that's fine.
   But my point is this other caching is much more difficult to do
   accurately because of buffer changes and assumptions about the
   filtering strategy.  For example, a cache that assumes that adding a
   character to the end of entity will always produce a subset of
   previously obtained completions will fail if the completion style is
   some kind of "regexp-based" thing, or if that character is '.', for
   example But it might work decently in some situations.

Anyway, another generic point that I've been trying to make is that
filtering/completion-styling should be done as close to the source of
completions as possible.  In Elisp this is easy to do (completions are
basically lists of strings or the big cheap-to-access obarray).  If the
source of the completions is removed from Emacs by some latency, the
experience is never going to be as good as Elisp.

- If you want to fully honour `completion-styles` which is an Elisp
  facility, you need to know most complete set of completions possible at
  all necessary times.  That's going to be slow (but read my final
  paragraph).

- If you're OK with letting the server do the filtering and the
  highlighting, you can make a "backend" style like I did for SLY, for
  example.  It's going to be faster, but `completion-styles` won't be
  honoured.  That's doesn't mean you give up 100% on "flex".  In SLY,
  there is flex implemented on the Common Lisp side, and for Eglot, many
  LSP server do their own flex matching.

In any case, it is generally possible to design responsive backends
systems that let the user interact with Emacs while the "slow requests"
are ongoing, by having these backends cancel themselves when input is
available.  This uses while-no-input and sit-for.  See jsonrpc.el's
jsonrpc-request for an example.  I've been using these kinds of systems
very successfully over the past 4-5 years with SLY, Eglot as backends
and Company UI as a frontend.

Hope this helped,
João