RE: predictive.el -- predictive completion of words as you type inEmacs

[Phillip Lord]

Toby 'qubit' Cubitt wrote:
> 
> Wish I'd found pabbrev.el when I was first looking for such a package!
> I would probably have contributed code to it instead of writing my
> own. But maybe it didn't exist at the time. (I was looking about two
> years ago, and predictive.el has been around for almost that long. I
> just wasn't confident that it worked well enough to post it to
> gnu-sources before now).


My svn log tells me pabbrev is about 3 years old. 

I always try to post to g.e.s early because someone tells you that
it's already been done! pabbrev was an exception to this rule. 

 
> 1) "Philosophically", predictive mode treats its dictionaries more
>    like static reference sources (although they are obviously updated
>    as it learns), whereas pabbrev mode treats them more as a dynamic
>    analysis of buffer contents. 

Yep. 

 
>    The disadvantage is that dictionaries have to be created as a
>    separate step before using them. The advantage is that all words
>    in the language are always available for completion, they don't
>    need to already have been used in a buffer.

This is true. You can do "pabbrev-scavenge-buffer" to get all the
whole thing at once. Idle cycle scavenging will continue. pabbrev's
lack of dictionaries also mean that it can work with any type of 
buffer. pabbrev is great in lisp or java code for instance. 

 
> 3) The pabbrev dictionaries aren't persistent between emacs
>    sessions. Predictive dictionaries are (though it is possible to
>    make non-persistent dictionaries). The advantage of persistent
>    dictionaries is obvious: word frequency information keeps
>    accumulating, so the dictionaries increasingly adapt to your
>    writing style.
> 
>    Of course, it wouldn't be that difficult to make pabbrev
>    dictionaries persistent, it just hasn't been done yet.

I did think of doing that. In the end, I stopped because I couldn't
work out how to persist hash tables (alists are easy as you can 
pp them out and read them back in. Doing this from and reading 
back to a hash is slow). 

In the end, I decided to leave this because it also means that 
I don't have to garbage collection dictionaries. Also, I figure, 
that the words you want to use most frequently, change over time. 
I don't want the word "ontology" popping up when I'm writing lisp
code, and I don't want the word "emacs" popping up when I'm 
writing a paper. 





> 4) Predictive mode can automatically switch between different
>    dictionaries in different regions of a buffer. For example, it
>    automatically uses a dictionary of maths commands within a LaTeX
>    equation environment, or of HTML tags after "<".

This would be a good feature to add to pabbrev, but one I am unlikely
to add. pabbrev generally works well in HTML because all the tags
start with "<" so you only get offered completions starting likewise.


> 
>    Along the same lines, I plan to interface predictive mode with the
>    semantic package 

Good plan.


> 5) There are differences in the user interface for predictive and
>    pabbrev mode. Some that are down to nothing more than default key
>    bindings (e.g. punctuation characters accept completions in
>    predictive mode). Others are differences in the features provided:
>    e.g. predictive mode can display completions in a tooltip or
>    menu, the most likely completions can be selected with single
>    character hotkeys.

Again, nice features. The UI of pabbrev is meant to be as small and as
fast as possible. This, for example, is the reason why I have abused the
standard conventions for minor modes and bound [tab] to something else. 
(If RMS is reading this, apologies--I normally pay attention to these
conventions, but there are exceptions to every rule. I have tried to
make
the impact on the normal UI as minimal as possible). 



> 
> 
> Low-level differences:
> 
>    I *think* that the predictive dictionaries have better space
>    (memory) scaling than the pabbrev dictionaries, with the trade-off
>    of the slower lookup (O(log n) instead of O(1)) described above.

I wouldn't be surprised. pabbrev is not space efficient. It doesn't 
need to be as it dumps dictionaries between invocations, and it only 
scavenges from the current buffer; finally, space requirements only go
up with unique words. Most people don't use that many words. 


> 
> 7) Predictive doesn't keep the words sorted by frequency. It sorts
>    them on the fly when they're looked up. This adds even more
>    overhead to lookup than pabbrev's method (though it does make
>    inserting words faster).
> 
>    It's amusing that both packages seem to have chosen the
>    "wrong" method, given their "philosophy". With it's more static
>    dictionaries, why does predictive sort on the fly instead of
>    storing that information? With it's dynamically generated
>    dictionaries, why does pabbrev have to keep them sorted in the data
>    structure? Of course, the answer is that lookup in predictive is
>    fast enough even when sorting on the fly, and word insertion in
>    pabbrev runs as an idle process so it too is already fast enough.

No, you have this wrong. Word insertion runs in the command loop, 
on the post-command-hook. The sorting, however, happens in the idle
cycle. Sorting makes the process of scavenging words for the dictionary
much less efficient, but I don't care, because it improves performance
with the interactive parts of pabbrev. A word insertion just does a hash
look up. Even the dialog doesn't require much (hash look up in O(1), and

print an alist O(m) where m is the number of suggestions (rarely more
than
10). 



> I've probably missed some things, since I haven't played with pabbrev
> mode much yet. I don't know how much the packages could benefit from
> each other. I dislike duplication, and would have preferred to
> contribute to an existing project if I'd known it existed. But trying
> to combine them into one package now doesn't look too likely.

I tend to agree. 


> The predictive user interface code could be useful for pabbrev
> though. For example, if you call the `complete' function from
> `predictive-completion.el'...

I might have a look at this. To be honest, though, I largely consider
pabbrev to be feature complete (although I've added a small feature
or two recently). It mostly "does the right thing" which I am pretty 
happy about. Complicating the UI is likely to make things worse. 

I would have quite liked to pull out the UI and the idle cycle code.
Both turned out to be a major pain, and others might like them. 
 
> Let me know what you think, and feel free to continue privately if
> this is getting too off-topic.


Good point. I don't care, but others might, so we should probably
continue
off list. 

Incidentally, you seem to be sending everything three times!

Phil