Re: Patch for unicode in varnames...

[L A Walsh]

George wrote:
> On Mon, 2017-06-05 at 16:16 -0700, L A Walsh wrote:
> > George wrote:
> > > On Mon, 2017-06-05 at 15:59 +0700, Peter & Kelly Passchier wrote:
> > > > On 05/06/2560 15:52, George wrote:
> > > > > there's not a reliable mechanism in place to run a script in a 
> > > > > locale whose character encoding doesn't match that of the script 
> > > > From my experience running such scripts is no problem, but correct 
> > > > rendering it might depend on the client/editor. 
> > > It depends on the source and target encodings. For most pairs of 
> > > source and target encoding there is some case where reinterpreting a 
> > > string from the source encoding as a string in the target encoding 
> > > (without proper conversion) will result in an invalid string in the 
> > > target encoding. For instance, if a script were written in 
> > > ISO-8859-1, many possible sequences involving accented characters 
> > > would actually be invalid in UTF-8. 
> >
> > ---
> >     Um... I think you are answering a case that is different than
> > what is stated (i.e. locale being same as used in script).  So no
> > conversion should take place.
> >     
>
> Eduardo's patch ... can only work correctly if the character set 
> configured in the locale is the same as the character set of the script.
----
   Right.  The 1st paragraph (written by you), above, mentions that. 
Given the 1st paragraph (which no one is contesting), we are only
talking about the case where the run locale and script locale are the same.

 The Passciers wrote that regardless of such agreement, you can still find
editors that may be ignoring the locale r have no locale support at all
and only display characters where the editor was written.  While that is
also true, it can't really be helped: if your local editor only writes
in Chinese and the script is written in ASCII, you may be out of luck
in having it display properly.

> Broadly speaking I think the approach taken in Eduardo's patch 
> (interpreting the byte sequence according to the rules of its 
> character encoding) is better than the approach taken in current 
> versions of Bash (letting 0x80-0xFF slide through the parser) - but 
> that approach only works if you know the correct character encoding to 
> use when processing the script. The information has to be provided in 
> the script somehow.
---
   Not exactly -- as the only variable-length-encoding scheme that 
linux systems have had to worry about is UTF-8.  So if you encounter 
UTF-8 in the
input, it is probable that you can use UTF-8 for the whole script. 
Otherwise, use a binary decoding stream (letting 0x80-0FF) be
treated as a 2nd half of a 256-byte charset, OR a 128-byte charset
without the a parity bit stripped that is left "as-is".

   The utility "file" is one example of such a utility that can usually 
tell
the encoding type of a text file -- at least telling the difference between
UTF-8, ASCII and some 8-bit local charset.

   While such methods may not be 100% accurate, they are usually good 
enough
for most usages where one isn't running (we hope) random scripts of unknown
origin off the web.

   FWIW, I think we are in agreement, though it may not be clear!  ;-)