Re: Fix u32toutf8 so it encodes values > 0xFFFF correctly.

[Eric Blake]

On 02/22/2012 07:43 PM, John Kearney wrote:
> ^ caviot you can represent the full 0x10ffff in UTF-16, you just need 2
> UTF-16 characters. check out the latest version of unicode.c for an
> example how.

Yes, and Cygwin actually does this.

A strict reading of POSIX states that wchar_t must be wide enough for
all supported characters, technically limiting things to just the basic
plane if you have 16-bit wchar_t and a POSIX-compliant app.  But cygwin
has exploited a loophole in the POSIX wording - POSIX does not require
that all bit patterns are valid characters.  So the actual Cygwin
implementation is that on paper, rather than representing all 65536
patterns as valid characters, the values used in surrogate halves
(0xd800 to 0xdfff) are listed as non-characters (so the use of them
triggers undefined behavior per POSIX), but actually using them treats
them as surrogate pairs (leading to the full Unicode character set, but
reintroducing the headaches that multibyte characters had with 'char',
but now with wchar_t, where you are back to dealing with variable-sized
character elements).

Furthermore, the mess of 16-bit vs. 32-bit wchar_t is one of the reasons
why C11 has introduced two new character types, 16-bit and 32-bit
characters, designed to fully map to the full Unicode set, regardless of
what size wchar_t is.  It will be interesting to see how the next
version of POSIX takes the additions of C11 and retrofits the other
wide-character functions in POSIX but not C99 to handle the new
character types.

-- 
Eric Blake   eblake@redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

signature.asc
Description: OpenPGP digital signature