Re: BUG? RFE? printf lacking unicode support in multiple areas

[Eric Blake]

On 05/20/2011 02:30 PM, Linda Walsh wrote:
> i.e. it's showing me a 16-bit value: 0x203c, which I thought would be the
> wide-char value for the double-exclamation.  Going from the wchar
> definition
> on NT, it is a 16-bit value.  Perhaps it is different under POSIX? but
> 0x203c taken as 32 bits with 2 high bytes of zeros would seem to specify
> the same codepoint for the Dbl-EXcl.

POSIX allows wchar_t to be either 2-byte or 4-byte, although only a
4-byte wchar_t can properly represent all of Unicode (with 2-byte
wchar_t as on windows or Cygwin, you are inherently restricted from
using any Unicode character larger than 0xffff if you want to maintain
POSIX compliance).

> 
>> Since there is no way to produce a word containing a NUL character it is
>> impossible to support %lc in any useful way.
> ----
>     That's annoying.   How can one print out unicode characters
> that are supposed to be 1 char long?

I think you are misunderstanding the difference between wide characters
(exactly one wchar_t per character) and multi-byte characters (1 or more
char [byte] per character).

Unicode can be represented in two different ways.  One way is with wide
characters (every character represents exactly one Unicode codepoint,
and code points < 0x100 have embedded NUL bytes if you view the memory
containing those wchar_t as an array of bytes).  The other way is with
multi-byte encodings, such as UTF-8 (every character occupies a variable
number of bytes, and the only character that can contain an embedded NUL
byte is the NUL character at codepoint 0).

Bash _only_ uses multi-byte characters for input and output.  %lc only
uses wchar_t.  Since wchar_t output is not useful for a shell that does
not do input in wchar_t, that explains why bash printf need not support
%lc.  POSIX doesn't require it, at any rate, but it also doesn't forbid
it as an extension.

> This isn't just a bash problem given how well most of the unix "character"
> utils work with unicode -- that's something that really needs to be solved
> if those character utils are going to continue to be _as useful_ in the
> future.
> Sure they will have their current functionality which is of use in many
> ways, but
> for anyone not processing ASCII text it becomes a problem, but this
> isn't really
> a bash is.

Most utilities that work with Unicode work with UTF-8 (that is, with
multi-byte-characters using variable number of bytes), and NOT with wide
characters (that is, with all characters occupying a fixed width).  But
you can switch between encodings using the iconv(1) utility, so it
shouldn't really be a problem in practice in converting from one
encoding type to another.

>     That said, it was my impression that a wchar was 16-bits (at least it
> is on MS.  Is it different under POSIX?

POSIX allows 16-bit wchar_t, but if you have a 16-bit wchar_t, you
cannot support all of Unicode.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

signature.asc
Description: OpenPGP digital signature