Re: bidi reordering in program source buffers (was: bidi-display-reordering is now non-nil by default)

[Eli Zaretskii]

I've started a new topic, and suggest to have a separate one for each
issue that was discussed under the old name.

> From: Chong Yidong <address@hidden>
> Cc: Stefan Monnier <address@hidden>, address@hidden
> Date: Fri, 19 Aug 2011 16:00:30 -0400
> 
> Here is one example: in Emacs Lisp source code, I want every Lisp string
> to be treated as a single bidi segment (paragraph).

The example of strings in program source was on the table since the
beginning of this discussion, so it's obvious it should be supported.
I actually meant to have a discussion about HTML/XML buffers.

But even strings have some aspects that we never considered in enough
detail.  What about this use case (from C-like programming languages,
but similar problems exist with `format' in Lisp)?

   printf ("ABCDEF %d\n", foo);

If all we do is reorder the string as a single R2L paragraph, we get
this jumble:

   printf ("d\n% FEDCBA", foo);

which is simply illegible.  And if we reorder it as a single L2R
paragraph, we get this:

  printf ("FEDCBA %d\n", foo);

which is a bit better, but still wrong.  The correct display, that
corresponds to how this will look on any bidi-aware display, would be
this:

  printf ("%d FEDCBA\n", foo);

Note that the correct rendering depends not only on the bidi types of
characters in ABCDEF, but also in the characters produced by the
format specifier.  %d tells us exactly what these characters would be,
but what if we had a %s specifier, which could be anything?

Evidently, just following the cues from font-lock is not nearly enough
to DTRT even in this very simple use case.  For example, even if we
want to decide whether to reorder as R2L or L2R paragraph, font-lock
gives no clues.

Also, Stefan mentioned Lisp symbols.  Do we want to support symbols
that use R2L characters?  (One use case where it could be necessary
are those Lisp packages that store arbitrary words as symbols.)  If we
do, just following font-lock will flood any Lisp buffer with text
properties, with the net effect of slowing redisplay, even if none of
the symbols use R2L characters, which will happen in the absolute
majority of use cases.  Again, following font-lock sounds like a
design that doesn't scale well, and punishes the absolute majority of
use cases for the benefit of a tiny fraction of them.  That doesn't
sound right to me.