I've tested the algorithm on about 3000 words. I thought I'd fix the learning part and test it before moving on. Does Ibus use the same suggestions file as libvarnam, that is, the one at $HOME/.local/share/varnam/suggestions/ml.vst.learnings? Because for some reason the words that are present in ml.vst.learnings don't show up as suggestions in Ibus. And deleting the ml.vst.learnings file don't seem to affect the suggestions at all - old words that I typed in using Ibus was still suggested.
Also, can we have a code review later this week? I think it would be better to clean what I've done so far and have something solid before fine tuning it.