bug#37036: [PATCH] Inconsistent ASCII and Latin char categories

[Mattias Engdegård]

15 aug. 2019 kl. 21.23 skrev Eli Zaretskii <eliz@gnu.org>:

> I think they should become entirely useless, i.e. we should stop using
> them.  We have the entire Unicode database with all the character
> properties for quite some time now, and should favor using that
> instead.  Categories are an old kludgey hack, which goes back to
> pre-Unicode Emacs; it can never be anything but arbitrary, and we will
> never be able to fix that anywhere near completely.

Thank you, I see what you mean, and I agree that Unicode properties probably 
are better for most purposes.
In any case, I wasn't aiming for perfection; that is indeed a fool's errand. It 
was just a discovery of a rather obvious mistake, and evidence of code that 
doesn't work properly because of it. I thought the patch would be rather 
uncontroversial.

>> Consider the function fill-polish-nobreak-p. It is clearly written with the 
>> assumption of a reasonable definition of the Latin category, and it doesn't 
>> work as expected because of that.
> 
> Can you tell the details of where this function doesn't work?  I'd
> like to understand why fixing it needs to change the categories.

Right: it attempts to match a single-character word before point, with the 
assumption that \cl would match any Latin(-script) letter. However, since that 
expression matches most of ASCII as well, the function incorrectly says that 
line-breaking would be disallowed after "In my dreams..." or "(She smiles!)" or 
"He died in 1951." (well, the equivalents in Polish).
Some details are in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=20871 .

Of course it doesn't require the categories to be fixed. The point is that if 
there is some code that doesn't work because of the broken categories, there 
may very well be more.

> I don't think we should fix those mistakes, because that's an
> impossible goal.  We should instead gradually stop using categories
> for anything serious, certainly for any new code.  We should use the
> UCD properties and the various char-tables built upon that instead.

Perhaps, but categories still have one thing going for them: they have fairly 
good regexp support.