Re: XeTeX encoding problem

[Gavin Smith]

On 15 January 2016 at 00:11, Karl Berry <address@hidden> wrote:
>     it means that you want to use native UTF-8 support in my humble opinion.
>
> Not necessarily.  The problem isn't encodings, it's fonts.  The two
> things are intimately and fundamentally tied together, and that cannot
> be escaped.
>
> By switching to native UTF-8, the support in texinfo.tex for characters
> outside the base font is lost, as far as I can see.  Yes, you get some
> characters "for free" (the ones in the lmodern*.otf fonts now being
> loaded instead of the traditional cm*) but you also lose some characters
> (the ones that aren't in lmodern).

That's quite a major problem, I think. I didn't realise that so many
characters would be missing - this negates much of the benefit of
using native Unicode support. Is there really no font that aims to
include every single Unicode character?

>     (something like ``Table of Contents'' broken etc.)
>
> That can be fixed in other ways, without resorting to native UTF-8.

I agree.

>     CJK characters can not be used without native UTF-8 support.
>
> They still won't work without loading a font that has them (at the right
> time, without interfering with other fonts already loaded, etc.).  Not
> simple.  There are no CJK characters in lmodern, unless I'm totally
> missing them.
>
> Trying to write code to automatically load fonts per Unicode block (or
> some such) seems like a horrendously difficult problem.  Nothing in the
> TeX world has solved it, to my knowledge.

I don't see why it should be difficult; that said I haven't tried, so
aren't aware of all the complications.

> Anyway, it's up to Gavin whether to install your patch.  I don't have
> strong feelings about it.  Just pointing out that there are both gains
> and losses.

It would be fine as an option. If it's substandard in its glyph
support there's always the chance of improvements later.

That said, if there's a fix for the table of contents issue, maybe the
desire for native UTF-8 support will go away.

I don't think we should use my previous idea of only using native
UTF-8 support if "@documentencoding UTF-8" is not given. I thought it
was a neat idea but I can see that some people would find it
confusing.