Re: EOL: unix/dos/mac

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <address@hidden>
> Cc: address@hidden,
>     address@hidden,
>     address@hidden
> Date: Wed, 27 Mar 2013 03:12:11 +0900
> 
> Eli Zaretskii writes:
>  > > From: "Stephen J. Turnbull" <address@hidden>
> 
>  > > Currently NLFs *are* displayed, if they don't match the default for
>  > > the buffer.
>  > 
>  > No, they are displayed because nothing other than a single LF is
>  > treated like NLF by the Emacs internals.
> 
> Emacs doesn't get to define NLF; it's a Unicode concept.

Can we be less pedantic, please, just to have the water less muddy?

OK, let me rephrase: they are displayed because nothing other than a
single LF character is currently treated by Emacs as an end of line.
An end of line is never displayed by Emacs or sent to the screen, not
even on a TTY; it is acted upon by moving the display to the next line
(a.k.a. "new-line function").

> Those *are* NLFs, and in
> the "CR in *-unix buffer" form they *are* displayed as "^M"s, while in
> the "bare LF in *-doc buffer" form they *do* appear as stair-stepping
> lines.

I guess you meant "-dos", not "-doc".  Anyway, there are no
stair-stepping lines in Emacs because of this, because Emacs never
outputs the EOL sequences to the screen.  That is why the -unix or
-dos variants are meaningless in terminal-coding-system.

> "Everything" is of course an exaggeration.  At a minimum, you need to
> change delete and motion commands to handle the fact that EOL doesn't
> have a constant width in characters.  Should users be able to move
> *into* a CRLF in -unix buffer?  How about a -dos buffer?

No and no (and there won't be any -unix and -dos buffers under this
mode of operation).

> Should forward-char-command move into or *over* a CRLF?

No.

> Does it matter what the EOL convention is for that buffer?

No.

> What are we going to do for the occasional user who wants the less
> usual behavior for some reason?

What "less usual behavior"?

> You need to decide what (insert "\015") means in a -dos buffer

No decision required: it will insert an CR, like it does today.  If
that CR happens to precede a newline, it will become invisible when
inserted.

> Until now, skip-chars-forward and regexps would find EOL if the
> string defining the target contained "\n".  Is that going to
> continue to be true?  How do you propose to find a bare LF -- are we
> going to make users use octal or hex escapes, or do we define new
> string syntax?

I see no serious problems with this, sorry to disappoint you.

> Ah, but this is completely a different story.  Those there are C
> macros, and not visible to Lisp programs, which know that a line break
> is represented by a single character, U+000A.  That's no longer true
> for NLF, which by definition is composed of one or more *characters*,
> not code units.  It's *Lisp* code that has to deal with this.

Lisp code already needs to deal with similar complications, e.g. when
it moves across invisible text or text covered by a 'display' property
or overlay string.

>  > > Any code handling old-style hidden lines (with CR marking
>  > > "invisible" lines) will have to be changed.
>  > 
>  > First, we want to deprecate and remove this feature anyway (there's
>  > already an implemented alternative).  And second, we already handle
>  > this today so that we don't display ^M there; the same method can be
>  > used for the other NLFs.
> 
> Sorry, that breaks immediately.  That ^M is now an NLF, and you either
> treat it that way and not as an invisibility marker, or the meaning of
> the buffer changes when you switch that mode on and off in a very
> delicate way.

No, it doesn't break, like it doesn't today.  When selective display
is in effect, a buffer-local variable says that, so you can treat ^M
accordingly.

> So you can fall back on deprecation.  Has the feature actually been
> scheduled for deprecation and eventual removal?

Yes, long ago.

>  > What it adds doesn't seem so frightening to me, certainly less so
>  > than, say, adding bidi support ;-)
> 
> Agreed, but irrelevant.  bidi is a new feature necessary to support
> some languages currently used by millions of people, and the hairiness
> is mandated by UAX #9 -- an alternative implementation is not going to
> make conformance much easier.

You are missing my point, which was about implications _on_Emacs_ of
adding bidi support.  UAX#9 cannot (and didn't) help making design
decisions in that regard.

>  > The internal representation is still exposed, so nothing's changed in
>  > that department.
> 
> I know, and taking advantage of that exposure still falls in the class
> of "Kids, these stunts are performed by trained professionals.  Don't
> try this at home!"  Can you deny that?

No.  But I'm saying that given that exposure, the abstraction _will_
leak, and when it does, users will be unhappy again.

> I know that what you're talking about will break some invariants for
> character motion and editing at line end, and that worries me.
> Proof?  You're right, I have none.

You don't need a proof, because I agree.  But we already have quite a
few features that introduce peculiar effects into character motion,
and they didn't cause any catastrophes.  I don't see why this one is
any different.