Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Yes, but we are not talking about the internal handling.

Of course we are.  Whether explicit directionality is represented in
the buffer as directional marks or as a text property is a purely
internal matter.

 > Nor is it specific to text external to Emacs: after all, the
 > internal storage of text in Emacs, as in many other applications,
 > is just a linear byte stream.

That's just saying that you *can* use directional marks for this
purpose.  It is not a reason you *should*.

 > Such unexpected conversions behind user's back proved to be an
 > annoyance, as the experience of MULE shows.

With all due respect to Handa-san and the other MULE implementers,
MULE was implemented to be too smart for its own good.  This is a good
idea for Japanese text, where at this very second I can lay my hands
on files containing no less than 5 major very different encodings of
Japanese (not to mention at least 5 more minor corporate variations,
plus Japanese written in scripts meant for Chinese).  It was not such
a great idea for other environments, although it mostly doesn't hurt
too much.

I would not extrapolate from the MULE experience to designs that could
be a lot more precise and explicit.

 >   . since each character can have only one value of the direction
 >     property, you cannot do this in any simple way; you'd need to
 >     split the original region in 3 parts, which is ugly

It's no uglier than inserting control sequences, which means that text
has yet another feature that prevents it from being an array of
characters.

 >     and complicates what needs to be done when text in this region
 >     is deleted (keep in mind that the UBA mandates support of up to
 >     60 levels of such embedded direction reversals, don't ask me
 >     why, and Emacs is in full compliance)
 > 
 > > If for some subtle reason, they don't quite nest correctly for this
 > > purpose, overlays most likely will.
 > 
 > Overlays don't get copied with the text, so if you copy/paste text
 > into another area of the same buffer or into another buffer, the nice
 > display will be lost.

Ah.  I see.  I forgot about that "feature" of the implementation in
Emacs.

 > We could complicate the heck out of yanking so it reinserts the
 > overlays, of course, but why complicate things if an easier way is
 > available that is straightforward?

Because it's correct, mostly.  You have convinced me that it would be
quite inconvenient to implement these things in Emacs without
introducing XEmacs-like extents that can have some of the properties
of both text properties and overlays.  I'm still not convinced that
the problems Lars worries about won't annoy the users.

 > The question that bugged us during the early stages of the design
 > was how do you ensure this without asking Lisp application
 > programmers to jump through the hoops every time text is copied or
 > saved or read.  It turns out that using the directional control
 > characters is the easiest way.

That's reasonable.

 > If we drop the marks on yanking, text will look differently when
 > yanked, sometimes completely differently, to the degree of being
 > incomprehensible.

Obviously.  Nevertheless, I suspect there will be applications where
that is desirable (especially for marks at the boundary of the yanked
region).