Re: Emacs completion matches selection UI

[Toby Cubitt]

On Wed, Jan 08, 2014 at 03:23:21AM +0400, Dmitry Gutov wrote:
> On 07.01.2014 07:32, Toby Cubitt wrote:
> >> Examples:
> >> 1. `prefix-replaced' and `common-substring' both seem to be tailored to
> >> the dynamic interface.
> >
> > Not at all. `prefix-replaced' tells you that the buffer substring
> > that is being completed has already been replaced with new text in
> > the buffer. This is important for non-prefix completion (e.g. pattern
> > matches).
> 
> > I'm not sure if it's generally applicable.
[...]
> > Any UI that modifies the buffer text will almost certainly need to set
> > `prefix-replaced';
> 
> I don't really see why a non-prefix completion UI (or any other one) has 
> to modify the buffer text. Company has an inline "preview" frontend, but 
> it uses an overlay, and so the buffer text remains unmodified.

Ah, you mean using the `before-string' or `after-string' overlay
property? That does sound like a better implementation.


> AFAIK, neither Company nor CAPF keep this kind of history: if the
> buffer was modified, what happens after is a new completion, or no
> completion at all (if circumstances told us we should abort).

Neither Company nor CAPF support refining the completion list by adding
more characters to the prefix/pattern (somewhat similar to
isearch). Completion-UI does, and it's very useful. Indeed, this is one
of the desired features Ted explicitly asked for in his recent post.

In general, this may require storing a bit more history about the
completion process, so you can continue from where you left off
correctly. At the very least, you need to store which completion backend
was used (which I think Company does too; CAPF makes this trickier, since
it doesn't have a simple way of identifying completion sources).

You've convinced me that prefix-replaced isn't needed, as long as UIs
aren't allowed to manipulate the buffer text. I need to think about it
more, but perhaps that could be a good restriction to impose as it
prevents UI bugs from mangling the buffer text.


> > And the core Completion-UI code that ties all the UIs together makes
> > heavy use of the `prefix-replaced' property. So even in Completion-UI
> > it's not only used by the dynamic interface.
> 
> Could you explain how?

I didn't impose any restriction on the UIs in what they're allowed to do
with the buffer text, so the core uses this e.g. when figuring out what
text it should leave behind in the buffer when a completion is accepted.

But I think imposing a restriction that UIs don't modify the buffer text
during the completion process is probably a good one, in which case it
may be unnecessary.

> > Similarly, `common-substring' demarcates the longest common substring if
> > completion text has been inserted in the buffer, which is information a
> > third-party UI might very well want to make use of.
> 
> Why? How?

Who knows how someone might want to make use of it in their UI? They
might want to highlight the common substring differently, say. But this
one's just for convenience anyway. A UI can recalculate it from the
completion list in any case.

> > The properties documented in the `completion-ui-register-interface'
> > docstring are those that have a standard meaning in Completion-UI. Note
> > that nothing prevents a particular UI widget from storing its own data in
> > its own widget-specific overlay properties if so desired.
> 
> Sure.
> 
> >> 2. `completion-source' seems to have to refer to a Completion-UI
> >> source, right?
> >>
> >> So how would one use an UI defined as such, in a different package?
> >> Would Company define itself as a new source? Or add a source per each
> >> backend?
> >
> > Depends on what the generic Emacs API for specifying completion
> > sources ends up looking like.  If we end up using c-a-p-f,
> > `completion-source' will need to store something that identifies the
> > c-a-p-f entry that returned the completion candidates for this
> > completion process.
> 
> Hmm, I was expecting something easier, since IIRC you said Company 
> should have been able to use it without major changes.

Maybe I misunderstood what you were asking. What I meant was, until we've
settled on the API for completion sources, we can't decide on the format
of the data that records which completion source ended up being used.

If you meant what would Company have done in the *current* Completion-UI
implementation, then yes: Company would have simply defined its backends
via the Completion-UI API, instead of via CAPF or company-backends.

But we seem to heading towards a different API for defining and selecting
completion sources. Probably CAPF, but perhaps something more similar to
company-backends.

If the former, we'll very likely want to record which CAPF source
returned the completions somehow. If the latter, then the equivalent of
the backend name is fine. (In the current Completion-UI implementation,
this property stores the completion source name, which modulo
implementation differences is directly equivalent to your backend names.)

I think you're reading way too much into the details of the current
Completion-UI implementation. The details will inevitably change if we
integrate it into part of a generic Emacs completion API. But whatever
the specifics of the data stored, and whatever we call the
variable/property that stores that data, we'll still need some way of
recording which completion source returned the completions we're dealing
with. Without this information, a number of features become tricky or
impossible to implement in the UI. As you surely know, since both Company
already records and uses information about which backend was used, just
like Completion-UI does.

> > If we end up using c-a-p-f, `completion-source'
> > will need to store something that identifies the c-a-p-f entry that
> > returned the completion candidates for this completion process.
> 
> Probably just the value returned by the successful completion function: 
> start, end, collection (aka completion table, which is often a function).

But how would we use that in the Customization interface, to allow users
to customize the UI in different ways for different sources?

It seems useful to have this feature, so users can make use it if they
want to. (See the use-case examples I gave in my reply to Stefan.) I'd be
sorry to lose the per-source customization feature. It's one I make light
but frequent use of myself. And users are bound to ask for something like
this later!

It's a bummer that CAPF doesn't associate unique names with completion
sources, in the way that Company and Completion-UI do.

One possibility might be to have CAPF functions optionally set a :name
property in the optional PROPS return value if they want to support
per-source UI customization for that source. What I don't see at the
moment is how Customize would discover what all the possible CAPF source
names are, in order to conveniently list them as options in a menu. (In
Completion-UI, this is one of the things handled by the macro used to
define new sources.) But that's a detail.


> >>> The best comparison one can make is that
> >>> company-backends loosely corresponds to a mixture of
> >>> completion-ui-source-definitions and auto-completion-source-functions.
> >>
> >> The latter: yes, the former: no. It's never used the same way as you use
> >> `completion-ui-source-definitions'.
> >
> > I said "loosely corresponds", and I stand by that. One of the things a
> > `company-backends' entry does is to define when a given completion source
> > should be used. In Completion-UI, that part of company-backends' role is
> > performed by auto-completion-source-functions.
> 
> Yes, that's why I agreed with this comparison (and also made it earlier, 
> IIRC). I disagree with the other comparison.

Sorry, my bad. I read what you wrote the wrong way around.

> > Could well be. But you seem to be arguing both ways. You also argue that
> > the c-a-p-f API is bad because it's opaque and hard for users to
> > configure. If backend definitions and selection logic are always supplied
> > by package authors, then the fact that c-a-p-f is opaque isn't so
> > significant.
> 
> The opaqueness in c-a-p-f is bad because the exact values of 
> `company-backends' and `completion-at-point-functions' are significant. 
> A third-party package author can only push a function in either of these 
> lists, but they can be responsible for the eventual order of the elements.
>
> And the order matters, because it influences which completion backend 
> will get picked when several are suitable. When we get to the grouped 
> backends, which I've mentioned several times, which Company supports, 
> and CAPF will hopefully support in the future, being able to understand 
> and maybe change the values of either list becomes even more important.

I completely agree.

How do you suggest we could improve it, without replacing CAPF or
breaking backwards compatibility?

This is something you care about more and have more experience of in
Company than I do with Predictive. I'm pretty confident that I can adapt
the Completion-UI code to whatever API we settle on for defining
completion sources and selection logic, without too much pain.

> >> Please take a look at `company-etags' and `company-dabbrev' and see if
> >> you can point out the situations when the user might find the `prefix'
> >> code of either too limiting.
> >
> > dabbrev and etags are both sources that, if they're useful in a given
> > buffer, they're useful in the entire buffer.
> >
> > Consider the LaTeX math mode source, and the LaTeX environment name
> 
> My question was, if you could point out problems with any of our bundled 
> backends (or, failing that, third-party ones). If they look okay, maybe

Did some text get cut off here in your reply?

> > source, and the LaTeX preamble source, etc. Trying to code the selection
> > logic for all of these using the Company API looks very awkward to me,
> > compared to the simplicity of setting a few buffer-local variables in
> > Completion-UI.
> 
> If performance is the problem, we could solve that by either:
> 
> 1) Adding some pre-completion hook which would allow you to run some 
> code once, set a buffer-local variable, which all backend functions for 
> LaTeX can refer to later in `prefix' action.

I don't think this can work, because the choice of source depends on the
location of point and the text in the buffer, which changes from
completion to completion.

For the Predictive LaTeX support, I use a mish mash of piggy-backing on
jit-lock where possible, some regexp match tests, and a limited form of
incremental parsing of the LaTeX code when the rest aren't enough. (I
used to exclusively use the latter, implemented using my auto-overlays
package. Surprisingly this was more than efficient enough to cause no
noticeable delay when typing and auto-completing, even on very old
hardware. But writing and debugging the incremental parser definitions
were a nightmare, largely because TeX doesn't have a well-defined
grammar. So I replaced as much as possible of the parsing with simpler
methods.)

This kind of thing is simpler to do in proper languages (assuming you
already have an incremental parser like CEDET, or a framework like
nxml-mode).


> 2) Create a special "merged" backend that would collect the results from 
> all LaTeX backend in a carefully defined fashion.

This effectively means moving the selection logic out of Company and into
Predictive. Which isn't necessarily a bad solution. It's how I used to do
it in Predictive, until I generalised the selection logic and moved it
into Completion-UI so other sources could make use of it.

If we stick with the CAPF API, I suspect I'll end up moving my source
selection logic back into Predictive, and making it Predictive-specific
again.

> 3) Indeed add some hook analogous to `auto-completion-source-functions'. 
> But yeah, CAPF already does that.
> 
> > Sure, I could move that logic into Predictive itself, and have a single
> > Predictive LaTeX backend. But that serves to demonstrate that the API
> > isn't flexible enough to let me do what I want easily. Other markup
> > languages and programming languages make similar demands on the API.
> 
> We bundle several backends, some of them for programming modes, and so 
> far you haven't pointed out specific problems with any of them.

Well, so far none of the existing Company backends are trying to do
auto-completion from a dictionary of 100,000 words, in a language with no
regular grammar, and where even tenths of a second lag are sufficient to
make typing unusable.

It would be an interesting experiment to stress-test the selection
mechanism by reimplementing Predictive's LaTeX support in Company. But it
would take some effort to code, and I don't have time (nor really the
interest) to code this up, especially when the Emacs API for this is
likely to look quite different.


> >> Or you can define new backends that would do some common checks in
> >> `prefix' (maybe calling an extracted function with common code) and
> >> simply delegate all other actions to the respective base backend.
> >> Implementing this is trivial.
> >
> > And once you've finished doing this, and factored out common selection
> > mechanisms like regexps, faces and text properties into utilities
> 
> Regexps we have already (company-grab-...),

How fast is this if you have to go through, say, 100 moderately complex
regexps checking for a match? (See the predictive-latex.el in the old
Predictive tarball release for examples.)

> instead of faces one should be using `syntax-ppss', font-lock isn't
> always available, and text properties... hmm.

Font-lock faces are *very* useful when you care about efficiency, because
of all the effort that's gone into optimising jit-look. (Font-lock has
very similar speed constraints to Predictive: it must not get in the way
of typing.)

> > functions...you'll have reimplemented something closer the Completion-UI
> > API or c-a-p-f :)
> 
> Only if we add a similar hook, see 3) above.
> 
> >> I understand the principle, really. But the more one "cleanly separates"
> >> code, the harder it can be sometimes to read it, to get the full picture.
> >
> > Indeed, which is why I listed grouping the completion and selection logic
> > into one place as one of the things I liked about Company's API.
> >
> > Perhaps the cleanest and most flexible solution would be to have a list
> > with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
> > COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.
> 
> I believe this suffers from the LaTeX problem you've described above. If 
> you have a dozen of completion functions for LaTeX, this scheme expects 
> you to have a dozen of corresponding test functions,

Indeed.

> and when one fails, the next will be called, and it won't be able to
> use the results of the previous call (unless they set and use some
> common buffer-local variable, which Company backends could also do; but
> that's ugly).

In my use-case, there's no useful information from previous tests anyway.

But I don't think this is such an important use-case to consider for a
generic API. Predictive has unusual requirements (always-on
auto-completion as the main use-case imposes extreme efficiency demands).
And (La)TeX and also Texinfo are odd cases, because their lack of a
regular grammar makes background incremental parsing more difficult
compared to programming languages.

Probably the best solution is to move the fast-selection logic I
implemented for use in LaTeX and Texinfo can back into Predictive.


> > And then supply a bunch of standard utility functions for use as
> > TEST-FUNCTION's, for testing regexp matches, faces, text-properties,
> > etc.
> 
> Like I described previously, such stand-alone tests probably won't be 
> very useful as values of this alist. One usually has to call several of 
> them to see if a completion-function is suitable.

I meant to write "for use *in* test functions". Simple utility functions
for testing regexps (which you already have), faces, etc. which can be
combined to build a test function.


> >> It makes certain amount of sense, although it looks like it could make
> >> creating a "merged" completion function more difficult.
> >
> > I doubt it'll be insurmountable. Also, merged completion functions are a
> > rather advanced feature that may not belong in core Emacs anyway (though
> > it would be good if the API supported them without ugly hacks).
> 
> It's a good feature enabling some kinds of backends that aren't usually 
> useful on their own (like Yasnippet). In my book, that's a good argument 
> to accept or reject an API.

I'd be very happy to see a solution in Emacs if you can come up with one
and convince people to merge it into core. As I said, adapting the
Completion-UI code to a different completion source API shouldn't be
difficult.

> >> We'll see.
> >
> > Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
> > and opaque c-a-p-f API replaced with something cleaner and simpler, I
> > don't see us winning that argument.
> 
> Personally, I'd probably be fine with c-a-p-f as long as it's powerful 
> enough. Moving to a less featureful API is likely out of the question, 
> but if it's demonstrated that c-a-p-f is fairly unsuitable for 
> implementing some features, I believe it would be a good reason to rule 
> it out.

In Completion-UI, I implemented combined completion sources through a
`completion-ui-combining-complete' wrapper function. (They're not
functionally equivalent to your merged sources, but the requirements are
somewhat similar.)

Would using something like that to build merged CAPF functions be a
solution for Company? I know this isn't particularly user-friendly if you
want to allow users to easily define their own merged sources. I don't
know if that's something people regularly do in Company.


As I said, it's not me you need to persuade if you want to replace
CAPF. I would be happy to see a nicer API than CAPF make its way into
Emacs core. But I also understand why there will be resistance from Emacs
devs to replacing an existing API unless there's a clear and pressing need.


> >> But the general approach, while flexible on its surface, complicates
> >> things if I intend to use any existing sources, written by third
> >> parties. Because their authors are unlikely to have anticipated the
> >> logic I'll add in my custom predicate function and to have written any
> >> code in their packages I might use. Or, at least, that's considerably
> >> less likely.
> >
> > I don't get your argument here. You have to wrap the third-party
> > completion function in Company in order to code the appropriate backend
> > selection logic. In Completion-UI, you put that code in an
> > `auto-completion-source-function' instead, and probably don't need to
> 
> Which code? There's likely to be none.
> 
> Hence, more effort required on my part.
> 
> Here I'm describing an organizational problem caused by an API. Not a 
> technical one.

I think we probably agree. Not worth pursuing this particular discussion
further really, since I'm not suggesting we use the Completion-UI API for
source selection.

There's one thing we should perhaps think a bit more about.

Is it right to say that the majority of the Company backends are selected
based on global properties of a buffer (e.g. major-modes)? This seems to
be the case for the default `company-backends' list. In Completion-UI I
was almost entirely focused on selecting backends based on local
properties at different locations within a buffer (regexps, faces,
syntax, etc.).

How do you envisage supporting local source selection in the new (or
enhanced CAPF) API? Would this kind of local completion source selection
always be implemented within a single backend, like the Company CEDET
backend? Or should there be a convenient way of supporting it in the
generic source selection API?

This kind of relates to your point about the ordering being important...


> >> Like mentioned above, delegating the search for completions to an
> >> existing backend is trivial. These are functions, and as such they are
> >> stateless. Just call (other-backend 'candidates current-prefix).
> >
> > But this results is convoluted and somewhat confusing code. If the
> > selection logic and completion function were separate, you wouldn't need
> > to use such hacks.
> 
> Looks straightforward to me. Function delegation is a rather simple concept.
> 
> We have examples of that in the Company frontends: take a look at 
> `company-pseudo-tooltip-unless-just-one-frontend' and 
> `company-preview-if-just-one-frontend'.
> 
> `company-dabbrev-code' also delegates to `company-dabbrev', but it just 
> uses a public function from that backend's package, which is also a 
> valid approach.

Following the code path is unnecessarily convoluted. I can't tell that
company-dabbrev-code internally delegates to the company-dabbrev backend
just from looking at `company-backends'. If it looked more like:

((use-dabbrev-code-p . dabbrev-completion)
 (use-dabbrev-p . dabbrev-completion))

it would be immediately obvious that these are two different ways of
using the same source of completions.

> > It understand why you're harping on about the limitations of the
> > default hook entries
> 
> I just can't see how you find them useful on their own. I'll shut up
> now. :)

Look at predictive-latex.el, which uses them on their own :)

But I'll shut up too. It seems clear to me from our discussion that we
should either use CAPF or something closer to Company for the completion
source selection API.

From a Completion-UI perspective, I don't much care what we
use. Switching between completion sources was never the main focus of
Completion-UI (the UI was the main focus). I should be able to adapt the
code to whatever we settle on. Most of it's abstracted into accessor
functions/macros internally, anyway.

> > Indeed, though this is starting to look a lot like defining some standard
> > hook functions for auto-completion-source-functions. As you make the
> > company-backends API more flexible and convenient, you'll increasingly
> > find yourself reimplementing equivalent functionality to that of c-a-p-f
> > and the Completion-UI API. If were to start simplifying the Completion-UI
> > API or c-a-p-f, it'll increasingly look more like the Company API. Maybe
> > the sweet spot is in the middle?
> 
> Maybe. I can certainly see myself adding a
> auto-completion-source-functions analog in Company, as an advanced
> feature.

The question for now is, what should go in the generic Emacs API? Vanilla
CAPF, perhaps with some additional standard PROPS properties? If the
generic Emacs API can't replace company-backends in Company, then in my
view the API is no good.

You're the expert here :)


> >> Actually, I was thinking about the former option. Let's define widgets
> >> with an API in the usual sense, so it can be used by both
> >> `completion-at-point' and external packages. We'll need this kind of API
> >> either way, in order to be able to write new widgets.
> >
> > I'm confused. `completion-at-point' will never use an API for *defining*
> > new completion UI widgets (`completion-ui-register-interface'). It will
> > need to be modified to *invoke* the new UI widgets.
> 
> Here I'm thinking in terms of Company approach, I guess. The API for 
> defining a backend or frontend and the way it is used are the same: a 
> protocol consisting of messages and proper reactions or responses to them.
> 
> Thus, "widget has to respond to..." would be in the API for defining a 
> widget, and "widgets respond to..." would be in the usage API. 
> Essentially the same.

Sounds pretty close to the Completion-UI widget API. A widget has to
provide "function that responds to x", "function that responds to y".

> If you like to add convenience macros, extra infrastructure, etc, they 
> may diverge, but we'll need an API a completion package can use easily 
> use anyway, not just a bunch of private functions that can change will 
> every release.

Absolutely.

The way I imagine it, completion packages like Company would rarely need
to use the API for defining new widgets. (Unless they're defining custom
widgets that are so package-specific they don't belong in core Emacs. But
that seems like a rare case.)

What they *will* all use is the API for *invoking* the completion
interface, which we began to discuss below.

> > Copying and adapting some of the code from `complete-in-buffer' into
> > `completion-in-region' would suffice for that. Then
> > `completion-in-region' would remain the generic Emacs API for displaying
> > and selecting completion candidates (only now it would display them in a
> > nicer interface).
> 
> Maybe. At a first glance, `completion-in-region' will need access to 
> PROPS returned by completion functions, not just COLLECTIONS.

Agreed. I figured we could add an additional PROPS optional argument, as
this wouldn't break backwards compatibility...

> >> Yes, but see above. Using 1. from Company would be the current next step
> >> toward integration, as I see it.
> >
> > Really? Why should we use the Company UI widget API and interfaces in
> > Emacs, when the API is less flexible and less customizable than that the
> > Completion-UI widgetAPI, and it implements fewer widgets?
> 
> Sorry for being unclear. I meant the reverse: Company would use the new 
> widget API defined here, while still retaining its backends, at least 
> initially.

Thanks, makes sense now.

> Although while you don't own the popup widget, we do have one. Maybe 
> porting that code won't be too hard.

Great! It would be a good test of the API too, to see how easy it is to
port.

Sounds like your equivalent of the "dynamic" interface is better, too
(you avoid modifying the buffer text). I'd like to either port the
Company version, or adapt the Completion-UI one to use the Company
mechanism (whichever is simpler).


> > Did you really mean that you wanted to throw Completion-UI in the
> > garbage, rewrite Company yourself to be suitable as a generic Emacs
> > completion API, rewrite/port the missing UIs and features, and persuade
> > people to merge it into Emacs core? (If so, great! Let me know when
> > you're done, and then I can stop maintaining Completion-UI and switch
> > Predictive over to the new interface :-)
> 
> I'll think about it. :) Not sure about the "persuade people to merge in 
> into Emacs core", though.

Ah, but that's the hard part - the rest is easy ;-)

> But you could help kick this process off my filing an issue describing 
> Company's backend API shortcomings. Is it just the lack of 
> `auto-completion-source-functions'? Non-prefix completion?

Erm, I don't see any major shortcomings in the API. Probably that hasn't
come across in the discussion :-)

Minor ones that have come here were:

- No support for non-prefix completion (but it looks easy enough to
  extend the API).

- Difficult (impossible?) to distinguish between "no completions
  available" and "backend not applicable".


> > Probably I misunderstood what you wrote.
> 
> Yep. :)

Shame! I was hoping you were volunteering to do all the work ;-)

> > I only really have one strong requirement: if some form of
> > auto-completion mode gets included in the Emacs completion UI, I will
> > argue hard for it to be as powerful as Completion-UI's
> > `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
> > and `auto-completion-override-syntax-alist', and at the way all the UI
> > widgets are integrated.) Anything less powerful, and it will be useless
> > to me for Predictive.
> 
> I'll take a look, thanks.
> 
> >> Customizing hooks is a tricky business. I believe the opaqueness of
> >> c-a-p-f to the user is the main problem with the current API.
> >
> > But above you argued that backends and selection logic are supplied and
> > configured by package authors, not by users!? In which case package
> > authors can simply supply a c-a-p-f function, and users can add the
> > functions they want to c-a-p-f. (I believe Stefan made a similar point
> > earlier.)
> 
> Customizing a hook is tricky for user. Try `M-x customize-variable RET 
> find-file-hook'. You'll only see the buffer-local value, and not the 
> global one. But `run-hooks' uses both if the local value includes `t', 
> which it usually does.

I see the global value in Customize, even when I deliberately add a local
hook function. Showing the global value in Customize seems the correct
thing to do. Is that not what Emacs does? (I wouldn't know - I've never
Customized a hook. I always use `add-hook' in my .emacs for that.)

> Users can modify the hooks programmatically, of course, but that's a 
> step more difficult. And they'll also need to understand the values 
> already there, to be able to remove or rearrange them.
> 
> We'll also should discourage lambdas there. At the moment, my 
> find-file-hook contains this beauty, courtesy of autorevert.el:
> 
> #[nil "\302\301!\210\303\304!8\211\207"
>    [buffer-file-name auto-revert-tail-pos make-local-variable 7 
> file-attributes]
>    3]
> 
> We should have less of that.

:-)

> > Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you
> > seem to be arguing both ways.
> 
> Of course I'm arguing toward a middle ground comfortable to me. So: 
> package authors deciding when their backend is suitable to use -- good; 
> users deciding the order of trying backends and their groupings -- also 
> good.

OK, makes sense. If I understand correctly, CAPF is fine for letting
users decide the order of trying backends. It's the groupings part that's
less clear, right?


> > Swings and roundabouts. I could always add a
> > `completion-ui-major-mode-source' function that checks an alist, and then
> > it would be:
> 
> Yes, well, here you are discarding the "standard, tried-and-tested Emacs 
> mechanisms" of major mode hooks and buffer-local values. Which was 
> exactly my point.

Now I'm confused. I thought that was *my* point ;-)


> > - foo completion function
> > - bar completion function
> > - foo predicate function
> > - bar predicate function
> > - add two entries to the alist
> 
> You forgot adding `completion-ui-major-mode-source' to the relevant list.
> 
> > - code 1st bundled completion function and selection logic
> > - code 2nd bundled completion function and selection logic (which
> >    duplicates the logic in the 1st one, with different parameters)
> >    ...
> > - code 8th bundled completion function and selection logic (which
> >    duplicates the logic in the last 7, with different parameters)
> 
> Could that be a one-line function call, in each case with different 
> arguments? Aside from potential performance problems, it sounds rather neat.

Not really. The selection logic for each source involves a list of
regexps, faces, etc. to test, and these are different for each
source. Passing lists of hard-coded regexps and faces around would be
ugly.

I guess you could store the regexps etc. in configuration variables (one
variable per source, or a single alist), and just pass an identifier as
an argument.

> > - add 8 entries to company-backends
> > - arrange for package-specific sources to be added to company-backends
> >    dynamically
> >
> > Contrast with Completion-UI's, which involves a *lot* less coding:
> >
> > - register 1st source (no need to code a new completion function, just
> >    set one parameter when calling `completion-ui-register-source')
> 
> You can likewise delegate to the same completion function in Company 
> backends. It's the same amount of code (1 line).

But in Completion-UI, you don't even have to write the code - the macro
does it for you :)

> > It would be almost trivial to switch Completion-UI over to
> > c-a-p-f. All I'd need to do to replicate the existing functionality
> > is add a few additional properties to the PROPS return argument.
> 
> Great. I'd like to see the patch. :)

I'll start working on it as soon as we agree on the completion source
API, and as soon as I have some spare time. (Which unfortunately isn't
likely to be before the end of term here in Cambridge, as I have a
lecture course to prepare from scratch. But the feature-freeze makes it
less pressing for now.)


> > Presumably I could always set a buffer-local `company-backends' from
> > predictive-mode that only contains relevant backends if necessary.
> 
> Yes, of course.
> 
> > c-a-p-f is trivially customizable by users, as long as packages / Company
> > supplies suitable c-a-p-f functions for the backends.
> 
> When you say "trivially customizable", do you mean via the Customize 
> interface?

I just meant that it's a straight list of functions, so it's easy for
users to understand how to configure it.

I take your point about configuring hooks via the customization interface
being tricky. But at least configuring hooks is standard Emacs foo. I
suspect most users who want to change the order of functions in CAPF will
know how to set hooks.

It would be nice to define a Customization interface that lists the
available CAPF functions in a drop-down menu (whilst also allowing
free-form entries, of course). If I find a way of porting the
Completion-UI auto-updating customization interface magic to CAPF, this
would be feasible...


> > *Writing* c-a-p-f is non-trivial, but as you said earlier those
> > are/should be supplied by package authors. Or have I misunderstood?
> 
> I don't think I said that about c-a-p-f, but yes, they should be.

Good, we're on the same page.

> >>> 3. The most popular and useful "list-the-available-completions" UI is
> >>>      popup.el. Is there any chance of getting copyright assignment for
> >>>      this? Or will we be forced to code something equivalent from scratch?
> >>
> >> https://github.com/auto-complete/popup-el/issues/50
> >
> > Fingers crossed...!
> 
> I wouldn't hold my breath: IME, getting a response from that crowd on 
> non-trivial issues is hard. Collecting copyright assignments could be 
> harder still.

Shame. I guess we'll be recoding it then. At least it won't be from
scratch, since you've already got something similar in Company.

> P.S. My email client displays each new message from you as a separate 
> thread, possibly because your email address is timestamped, or maybe 
> because they don't contain a "References" header. Could you do something 
> about that?

Strange. Mutt threads them fine for me. The ones I've checked all contain
an "In-Reply-To" header. I haven't changed the mutt reply header
settings, so presumably mutt doesn't add "References" headers by default?

Perhaps your email client is ignoring the In-Reply-To header? (I somehow
doubt the date stamps in the tagged addresses are the issue. Threads
always contain messages from different senders, so it threading should
work even if your mail client considers the changing dated addresses to
be different senders.)

All the best,
Toby
-- 
Dr T. S. Cubitt
Royal Society University Research Fellow
and Fellow of Churchill College, Cambridge
Centre for Quantum Information
DAMTP, University of Cambridge

email: address@hidden
web:   www.dr-qubit.org