bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier]

> Sorry for the pause, let's continue this digression. Though perhaps it would
> be better moved to emacs-devel or somewhere else. ;-/

I see this API as an experiment.  I have no idea if I'll like
the result.  It's definitely far from being something ready to submit as
a proposal for a new design.

>>> As such, the most useful methods currently are: 1) Emacs Regexp, 2) asking
>>> server for whatever it thinks is suitable (the "backend" completion style).
>> For the backends: agreed.
>> For the frontends (i.e. `completion-styles`), `glob` is the more useful
>> one, I'd say (except for the "external" style, of course).
>> We might also want support for things like `or` and `and` patterns, but
>> I haven't managed to fit them nicely in that structure :-(
> Possibly, but which code would produce such patterns?

The `or` pattern?  No idea :-)
The `and` pattern?  Well, the `orderless` style, for one.

But indeed, I'm not sure it'd be useful to handle things like or/and
directly in there rather than by using union/intersection on
the resulting completions.  It's just an aspect of the design
I considered and I noticed that I had trouble extending it in
that direction.

Obviously, the caller which needs to collect a set of matching
candidates always has a choice between using a more refined pattern
or using a simpler pattern (including various calls with various
different patterns).

>>> I would also probably want to standardize on the recommended type of TO
>>> anyway: some of them are likely going to result in better performance
>>> than others.
>> The TO is chosen by the specific completion table, based on what it can
>> handle best.  So it should always be "optimal".
> Sorry, I meant the recommended type of FROM. Because if the original caller
> passes an arbitrary regexp, it will often get turned into a pair with
> a predicate where the latter calls string-match-p.

The caller should use the most primitive pattern they can.

> And if the type of FROM is standardized, there likely would be no need for
> a four-way bidirectional conversion. Maybe just a helper that converts from
> the original "main" type into any of the available that is
> currently required.

Indeed the default method does:

  (cond
   ((eq to (car pattern)) (cons nil (cdr pattern)))
   ((eq 'glob to) (cl-call-next-method))
   (t
    ;; Most conversions can be performed by going through `glob'.
    (pcase-let* ((`(,gpred . ,glob)
                  (completion-pattern-convert 'glob pattern))
                 (`(,tpred . ,newpattern)
                  (completion-pattern-convert to glob)))
      (cons (or gpred tpred) newpattern)))))

>>> So I guess it's also a way to make every completion table aware of PRED?
>> Note also that these `pred` patterns are expected to be exclusively
>> looking at the string (they're used for `completion-styles` kind of
>> functionality), so nothing like `file-directory-p` or `fboundp` kind of
>> predicates here.
> Would we consider those pred's "fast enough"?

I don't know.  I haven't had a use for a `pred` pattern yet, to
be honest.  As for the predicates returned by
`completion-pattern-convert`, they're currently just fancy booleans
indicating if the returned pattern is faithful or not :-)
So I'm not sure those predicates will survive this experiment.

> (benchmark-run 10 (let* ((re "yo[^o]*o")
>                          (completion-regexp-list (list re)))
>                     (all-completions "" sss)))
> ;; => 0.60s
>
>
> (benchmark-run 10 (let* ((re "yo[^o]*o"))
>                     (all-completions "" sss
>                              (lambda (s) (string-match-p re s)))))
> ;; => 1.14s

>>> That should work; though it might be hard to reach the same raw performance
>>> as the current all-completions + completion-regexp-list.
>> I don't see why: currently my code actually uses `all-completions` and
>> `completion-regexp-list`, so as long as the pattern can be turned into
>> a regexp without requiring an additional PRED (that's usually the case), 
>> <...>
> Right, I was talking about the possible exceptions.

I don't know what you're getting at.


        Stefan