[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#31679: 26.1; detect-coding-string does not detect UTF-16
From: |
Eli Zaretskii |
Subject: |
bug#31679: 26.1; detect-coding-string does not detect UTF-16 |
Date: |
Sat, 02 Jun 2018 10:42:22 +0300 |
> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Date: Fri, 01 Jun 2018 21:40:32 +0200
>
> I have been trying this (in real life the strings are often longer, of
> course):
>
> (detect-coding-string "h\0t\0m\0l\0")
>
> And I was surprised that this does not detect UTF-16 but instead gives
> (no-conversion).
First, you should lose the trailing null (or add one more), since
UTF-16 strings must, by definition, have an even number of bytes.
Next, you should disable null byte detection by binding
inhibit-null-byte-detection to a non-nil value, because otherwise
Emacs's guesswork will prefer no-conversion, assuming this is binary
data.
If you do that, you get
(let ((inhibit-null-byte-detection t))
(detect-coding-string "h\0t\0m\0l"))
=> (undecided)
Why? because it is perfectly valid for a plain-ASCII string to include
null bytes, so Emacs prefers to guess ASCII.
As another example, try this:
(prefer-coding-system 'utf-16)
(let ((inhibit-null-byte-detection t))
(detect-coding-string (encode-coding-string "áçðë" 'utf-16-be) t))
=> utf-16
but
(let ((inhibit-null-byte-detection t))
(detect-coding-string
(substring (encode-coding-string "áçðë" 'utf-16-be) 2) t))
=>iso-latin-1
So even when UTF-16 is the most preferred encoding, just removing the
BOM is enough to let Emacs prefer something other than UTF-16.
Morale: detecting an encoding in Emacs is based on heuristic
_guesswork_, which is heavily biased to what is deemed to be the most
frequent use cases. And UTF-16 is quite infrequent, at least on Posix
hosts.
IOW, detecting encoding in Emacs is not as reliable as you seem to
expect. If you _know_ the text is in UTF-16, just tell Emacs to use
that, don't let it guess.