Re: coding tags and utf-16

[Kenichi Handa]

In article <address@hidden>, Werner LEMBERG <address@hidden> writes:

> There is a serious problem with coding tags and utf-16 encodings of
> any flavour: Emacs simply can't recognize the tag.  This is a
> non-trivial problem.

Sorry for the late reply, but I think coding tag is useless
for a file encoded in some of utf-16 variants.

If a file has BOM at the head, BOM should tell the exact
encoding whatever is specified in coding tag.

If a file is encoded without BOM, we must use the less
reliable heuristics to guess utf-16be or utf-16le.  If you
find a coding-tag spec by ignoring all zero bytes at even
byte indexes, it means that the file is, in high
possibility, utf-16be whatever the tag value is.  If you
find a coding-tag spec by ignoring all zero bytes at odd
byte indexes, it means that the file is utf-16le whatever
the tag value is.

So, in any cases, a tag value itself is useless.  Then how
to detect utf-16 more reliably?  In the current Emacs
(i.e. Ver.22), I think we can use auto-coding-regexp-alist
or auto-coding-alist.  In the former case, we can register
BOM patterns and also something like "\\`\\(\0[\0-\177]\\)+"
for utf-16be.  In the latter case, you can use more
complicated heuristics in a registered function.

But, those are anyway just heuristics; not 100% reliable.
So I think we need a user option to turn it on and off, or
perhaps a user option to select which kind of heuristics.

---
Kenichi Handa
address@hidden