Re: Inadequate documentation of silly characters on screen.

[Stephen J. Turnbull]

Stefan Monnier writes:

 > I don't know what you mean.  The eight-bit "chars" were introduced to
 > make sure that decoding+reencoding will always return the exact same
 > byte-sequence, no matter what coding-system was used (i.e. even if the
 > byte-sequence is invaldi for that coding-system).  Dunno how XEmacs
 > handles it.

Honestly, it currently doesn't, or doesn't very well, despite some
work by Aidan.

However, I think a well-behaved platform should by default error
(something derived from invalid-state, in XEmacs's error hierarchy) in
such a case; normally this means corruption in the file.  There are
special cases like utf8latex whose error messages give you a certain
number of octets without respecting character boundaries; I agree
there is need to handle this case.  What Python 3 (PEP 383) does is
provide a family of coding system variants which use invalid Unicode
surrogates to encode "raw bytes" for situations where the user asks
you to proceed despite invalid octet sequences for the coding system;
since Emacs's internal code is UTF-8, any Unicode surrogate is invalid
and could be used for this purpose.  This would make non-Emacs apps
barf errors on such Emacs autosaves, but they'll probably barf on the
source file, too.

 > > And it should be either an error to (aset string pos 241) (sorry
 > > Alan!) or 241 should be implicitly interpreted as Latin-1 (ie, ?ñ).  I
 > > favor the former, because what Alan is doing screws Spanish-speaking
 > > users AFAICS.  OTOH, the latter extends naturally if you have plans to
 > > add support for fixed-width Unicode buffers (UTF-16 and UTF-32).
 > 
 > I understand this even less.

There's a typo in the expr above, should be "multibyte-string".  The
proposed treatment of 241 is due to the fact that it is currently
illegal in multibyte strings AIUI.

Re the bit about Spanish-speakers: AIUI, Alan is translating multiline
strings to oneline strings by using an unusual graphic character.  But
it's only unusual in non-Spanish cases; Spanish-speakers may very well
want to include comments like "¡I wanna write this comment in Español!"
which would presumably get unfolded to "¡I wanna write this comment in
Espa\nol!"  Not very nice.

Re widechar buffers: the codes for Latin-1 characters in UTF-16 and
UTF-32 are just zero-padded extensions of the unibyte codes.  I'm
pretty sure it's this kind of thing that Ben had in mind when he
originally designed the XEmacs version of the Mule internal encoding
to make (= (char-int ?ñ) 241) true in all versions of XEmacs.

 > I think XEmacs's fundamental tradeoffs are subtly different but
 > lead to very far-reaching consequences,

Indeed, but I'm not talking about XEmacs, except for comparison of
techniques.