Re: [PATCH] better isearch support for complex input methods

[Karl Fogel]

Dave Love <address@hidden> writes:
>  SM> Also it should be fairly easy to list
>  SM> the relevant charsets since I would expect that all the ideogram-based
>  SM> charsets are 2-byte charsets and we don't have thousands of
>  SM> those, do we ? 
> 
> It sounds as though categories are the correct tool for the job,
> though I'm not quite sure what the job is.

:-)

The job is to determine quickly whether a single character represents
a word or not, so forward-word can do the Right Thing.  

In English (etc), word boundaries are always whitespace or
punctuation, so forward-word's job is pretty easy.  In Chinese (etc),
word boundaries cannot be determined syntactically, but the most
useful approximation is to treat each character as a single word.
That almost always does what you want.

Note that it doesn't work to simply treat every character as its own
word in certain language environments.  The buffer may contain
mixed-language text, and we want forward-word to behave correctly in
moving across both Chinese characters and English words, even when
they're right next to each other.

Thank you for the suggestion about categories -- they seem like the
right solution, and they're even independent of the language
environment.  We probably don't even need a new category, we can just
use the `|' category, whose docstring says:

   "|: While filling, we can break a line at this character."

Based on the output of describe-categories, I think all the
ideograph-based languages already have this category set
appropriately.

scan_words() in syntax.c looks like the place to make this change,
does that seem right to you?  I'll work on it, not rushing, as we're
in feature freeze and Gerd has indicated that changes like this should
wait until after 20.1.

-Karl