Re: Emacs completion matches selection UI

[Toby Cubitt]

Hi Dmitry,

On Fri, Jan 03, 2014 at 12:20:38AM +0400, Dmitry Gutov wrote:
> On 31.12.2013 19:52, Toby Cubitt wrote:
> > I really wasn't trying to promote completion-UI over Company. Indeed, I
> > completely agree (as I tried to say in my previous post) that
> > company-mode is much more sophisticated than completion-UI, because it
> > aims to do something much more ambitious.
> >
> > Completion-UI was only ever intended as "plumbing": an elisp package that
> > provides functions and hooks to let you display and select completion
> > candidates in various UIs. Nothing more.
> 
> My impression was that you proposed it as a wrapper around Company 
> and/or related packages.

No, definitely not a wrapper. I was proposing using it as an Elisp
utility library. If anything, it's Company et al. that would build on top
of and wrap around a generic Emacs completion UI, rather than the other
way around.

> In that case, lowering the level of sophistication means lowering the
> amount of features, which is bad.

This thread was about including generic and more "modern" completion
user-interface(s) in Emacs. If this happened, I hope that Company and
other completion frameworks would make use of them, instead of each
rolling their own. Essentially, it would just move the generic parts of
the frameworks into Emacs. All the stuff on top of that that makes
Company more sophisticated would still need to be implemented in Company.

Providing a generic completion user-interface seems fairly close to what
I was trying to do when I separated out completion-UI from predictive
into a separate package (about 10 years ago now).

> But if you'd like us to just use the widgets from Completion-UI (and 
> maybe some code related to customizing them?), that would be okay, 

Yup, that's basically all I meant.

I imagine implementing Company on top of a generic Emacs completion UI
library would save you some maintenance effort. And a generic Emacs
completion UI would benefit from patches and improvements from all the
actively-developed Emacs completion packages.

Completion-UI *is* just a library providing completion user-interfaces,
plus a generic and somewhat sophisticated auto-completion-mode. It was
never intended to be a "completion framework", just a library of
integrated completion UI widgets and code. From where I stand, this seems
to be closer to what people were asking for in this thread.

I'm not pushing hard for Completion-UI inclusion. Just pointing out that
there's a lot of existing code out there, and it would make sense to
reuse it instead of reinventing the wheel for the umpteenth time. For
obvious reasons I know Completion-UI best, whereas I can't sensibly
comment much on the others. So I posted about what I know. I really don't
mind if people take that code from Completion-UI, or from Company,
auto-complete, anything, or any of the myriad others I've missed.

I'd hope that if and when Emacs included a generic completion UI, I'd be
able to declare Completion-UI obsolete and switch to using the standard
Emacs implementation in Predictive.

> although I'd have to look into fitting any of them inside the Company's 
> interaction with the command loop,

...which BTW is why I suspect the full Company framework isn't suitable
as a generic Emacs completion UI. 

Completion-UI doesn't need to touch the command loop. You pass a
completion function to `complete-in-buffer', and it displays the
completion candidates and lets the user select one, via the UI they've
configured through customize.

The only purpose of "registering" a completion function as a named
completion source, is to let users customize the UI separately for that
source (if they want to), and make it available as an choice in some
customization options.

The generic Completion-UI `auto-completion-mode' minor-mode is slightly
more invasive. It's not entirely clear to me at this point whether a
generic Emacs completion UI should include an auto-completion minor-mode
or not. I'm sure others will have opinions on this.

> > Company does *much* more than this. It's practically a way of life :)
> 
> It's not hard to describe Company in the same terms: it's a framework 
> designed to fetch completion candidates from the configured backends and 
> allow selecting between them in the configured frontends.

Sure, but as far as I understand (I could well be wrong - I've never used
it in anger) you use Company by enabling company-mode in your buffer as a
user, and this minor-mode changes how you interact with Emacs. Whereas
you use Completion-UI by calling its functions from Elisp when coding a
new Emacs package. It's really aimed at a "lower" level.

> The "show documentation" and "show definition" actions are small bonus
> features, as far as complexity of implementation goes.

Indeed. I only commented on them because you brought them up.

> > I considered switching to Company for the predictive-mode UI, since I'm
> > really not that interested in coding and maintaining user-interface
> > code. (It's both tedious, and difficult to do well.) But Company was too
> > all-encompassing: the UI elements weren't cleanly separated out from the
> > rest of the mode, and I couldn't see how to make use of them without
> > buying into the whole company-mode shebang.
> 
> Company front-ends are not independent widgets, indeed. They don't 
> encapsulate data, instead using the same buffer-local vars Company 
> operates with.
> 
> And the popup code is a relatively small part of the package, so 
> "switching to Company" really means writing a backend for it, in this 
> case. I wonder which part of the backend API you didn't like.
> 
>  > Maybe that's changed now.
> > Though I still can't seem to see a generic UI package separate from the
> > rest of company-mode.
> 
> No one's requested it, so far. There's the popular popup.el, and there's 
> dropdown-list.el of unclear providence on EmacsWiki which Yasnippet uses 
> optionally.
> 
> I've been thinking about splitting the Company popup code properly, but 
> being able to reuse (and contribute to) the popular option which is 
> popup.el would be better.

popup.el is great, but it's a way of simulating tooltips using overlays,
not a generic Emacs completion UI. (Though it would be nice if it was
part of it.)

> > I've got nothing at all against company-mode. Indeed, if I remember
> > right, someone (you?) has added predictive-mode as a company-mode source,
> 
> Nope.

Seems it was Joakim. My impression was Company backend definitions
belong in Company, so perhaps his code should be added there?

> > which is great! But I didn't want to force people to use company-mode in
> > order to use my predictive package.
> 
> The usual way to go about it is to have a core package (maybe using 
> completion-at-point-functions) and a couple of integrations with the 
> popular completion packages.

There is a core package: predictive. It's completely separate from
Completion-UI (even if the latter comes bundled with it in the
tarball). No doubt that's why Joakim could fairly easily write a
predictive backend for Company.

If Company backend definitions should live in the completion package
rather than Company, I could add it to Predictive instead.

> > I just wanted to provide simple UIs
> > for displaying and selecting completions.
> 
> I'd wager Predictive would have a better adoption if it wasn't coupled 
> to a yet-another completion package.

Well, I believe Completion-UI predates at least Company, anything, and
auto-complete. So "yet-another-completion-package" is misleading. Had
those packages existed when I coded Predictive, I doubt I'd have bothered
coding Completion-UI.

It's a shame Company et al. didn't build on and contribute to
Completion-UI, and instead chose to (each) reinvent the wheel. But that's
the usual messy history of code (which maybe we're finally going to do
something to tidy up?).

I'm really not bothered about pushing for widespread adoption. I write
the code because it's useful to me, I make it available in case it's
useful to others, and I respond to bug reports and feature requests since
dumping code on the web without maintaining it seems irresponsible. If
people want to use it as is, great. If people want to use it via Company,
great. It's not difficult to do the latter, as Joakim demonstrated. I
seem to recall someone integrated Predictive into anything, too.

> I, for one, don't even consider using it for this reason. Although
> maybe I'm just not the target audience.

I have no idea. Do you want fast predictive completion of plain text, or
text markup languages like LaTeX? If so, you're the target audience. If
not, then probably not. (Adding predictive code-completion features to
some IDE-style Emacs completion package like CEDET might be useful, but I
don't do enough hard-core coding to implement that myself.)

> > I'd be very happy to see the UI parts of Company stripped out and made
> > into a simple, generic completion UI package and added to Emacs.
> 
> Would you like to propose an API for it? Same as popup.el?

As a first attempt, I'd propose something like 

  (complete-in-buffer COMPLETION-SOURCE &optional PREFIX-FUNCTION)

where COMPLETION-SOURCE is a function to call to obtain completions, and
PREFIX-FUNCTION is an optional function to call to determine what to
complete.

Calling this would invoke whatever completion widgets the user had
enabled in customize (with sensible defaults - maybe the popup.el
interface or similar, or perhaps the default should be the usual
*Completions* buffer).

> > Since completion-UI was originally written as the UI for predictive-mode
> > (which is most useful in text modes), it's strong on plain text features,
> > and weaker on programming-related features. For example, last time I
> > looked completion-UI's auto-completion-mode was much more sophisticated
> > than that in any other package (which lacked many of the features that
> > are crucial to implementing predictive completion).
> >
> > That's why I think merging the best bits of the generic UI stuff from all
> > the various frameworks would be the best way to go.
> 
> It sounds good as an abstract idea, but I'd like to see a more specific 
> proposal.
> 
> >> Toby Cubitt <address@hidden> writes:
> >>> - has completely modularised completion user-interfaces, which can be
> >>>    used in any combination the user likes (within reason)
> >>
> >> You can have some of that in Company by setting `company-frontends' to a
> >> buffer-local value. Probably. I've never tried that, though, and I'm not
> >> sure if I'll ever want to, personally.
> >
> > Really? Some people like auto-displayed tooltips, some people hate
> > them. Some people like displaying completions in the echo area, some find
> > it a distraction. Makes sense to me (given that this is Emacs we're
> > talking about) to let people customize it the way *they* want via
> > customization options, with sensible defaults.
> 
> Sorry, I took it to mean that in Completion-UI you can configure 
> user-interfaces per completion source, which is probably not the case.

You can if you want to; you don't have to.

Customizing the UI per completion source very often makes more sense than
customizing it per buffer. If you're completing filenames, you probably
want the same interface in every buffer. Whereas you probably don't want
the same interface for both filename completion and for plain-text
predictive completion.

> The user can of course configure the used front-ends in Company, by 
> customizing the global `company-frontends' value (as opposed to 
> buffer-local ones I mentioned above).
> 
> > Yes. And a "completion browser" that organises all the completions into a
> > hierarchical set of menus. (As with most things, this generic version can
> > be overridden by particular completion sources to provide a specific
> > version that's more useful. I use that e.g. in the predictive-mode LaTeX
> > support when completing LaTeX commands, environments, labels, etc.)
> 
> The browser sounds useful, I'll have to look into it.

It seems to be popular, especially for completing from a large standard
set of keywords which you can't quite remember (as in LaTeX). At least, I
get complaints by email whenever I break it ;-)

> A mouse-only menu won't fit Company because it won't work for idle 
> completion, as I mentioned elsewhere in this thread, and we generally 
> want to provide a consistent keyboard interface. There'll also be issues 
> with implementation related to that.

When you register a new UI with Completion-UI, you tell it whether the UI
is suitable for idle completion. If it is, it becomes one of the
customization options you can select for the idle completion UI.

> >>> - lets you add a new completion UI with a single call to the
> >>>    `completion-ui-register-interface' macro
> >>
> >> Company allows you to do that with a handy macro called `defun'.
> >
> > Needlessly snarky.
> 
> Sorry. I was aiming for cute rather than aggressive.

OK, no worries!

>  > You still need some way to tell Company about the new
> > UI, so that it knows to invoke it.
> 
> Actually, no. After defining the backend function with `defun', the user 
> will need to add it to `company-backends', but that's not much different 
> from them having to customize `auto-completion-source' in Completion-UI.

Sounds like adding it to `company-backends' is more analogous registering
it with `completion-ui-register-source'. Customizing
`auto-completion-source' is something the user would do, not the Elisp
package coder.

In fact, it sounds like the two APIs are rather similar:

1. Define a completion function with defun (Completion-UI and Company)

2. Pass it to `completion-ui-register-source' (Completion-UI) or add it
   to `company-backends' (Company) to let them know about it.

At some point in the distant past, you used to just set a variable to
tell Completion-UI about a new completion function. It became a macro
call when I made Completion-UI more generic, and there needed to be a way
of optionally supplying additional information about what the completion
function does (e.g. if it does something other than standard prefix
completion), and of defining optional call-back functions, etc.

> > It's not a conversion at all. The sophisticated parts of company aren't
> > about displaying menus, or popups, or tooltips, etc. I'm just saying that
> > if you had a generic Emacs completion UI, you could (and should) rebase
> > Company's UI on that. This is a small and boring part of
> > Company. Wouldn't you be happy to see that included in Emacs, so everyone
> > could benefit instead of reimplementing the wheel?
> 
> Of course I would. But a) you don't own the popup code, which we would 
> like to use,

Indeed. Popup is one particularly nice UI. If it can't be donated, we'd
probably have to be reimplement it from scratch, which would be a
shame. Unfortunately, that's an issue however we decide to implement a
generic Emacs completion UI.

> b) it could be hard to augment Completion-UI widgets with new commands,
> such as, for example, which using Company you can currently press `f1'
> when the popup is active to pop up the documentation buffer, or `C-w'
> to see the function source. How hard would it be?

Trivial. In fact, it's already possible by overriding default functions
in the source definition. But it seems generically useful enough to be
worth creating a simpler API for.

> > That's why I went to the effort many years ago now of separating the
> > UI code out of predictive-mode into something generic. Unfortunately,
> > everyone then went off and invented new wheels (the UI code in
> > Company, anything, auto-compelte, etc. etc.).
> >
> >> Also, I think `company-backends' provides a nicer API than
> >> `completion-ui-register-source'.
> >
> > Could well be. So help design a good API for a generic Emacs
> > completion UI.
> 
> Yes, sure. Would someone maybe submit an initial proposal?

I second that. I'd like to see what people really want to see merged
before I'm likely to devote time to it. (Whether I *have* time to devote
to it is a separate question...)

Best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: address@hidden
web:   www.dr-qubit.org