bug#54488: 29.0.50; move-to-column/overlay-related regression in latest master, perhaps 28?

[Eli Zaretskii]

> Date: Tue, 22 Mar 2022 15:57:17 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 54488@debbugs.gnu.org, dgutov@yandex.ru
> 
> I think I see the problem.  Stay tuned.

I definitely see a problem with display strings with embedded
newlines, but I don't think I understand how it affects your use case.

Try this simple recipe:

  emacs -Q
  M-: (overlay-put (make-overlay 91 100) 'display "FOOBAR\nBAZQUUX") RET

You should then see this in *scratch (I indented by 2 columns, so it
stands out):

  ;; This buffer is for text that is not saved, and for Lisp evaluation.
  ;; To create a fileFOOBAR
  BAZQUUXt it with C-x C-f and enter text in its buffer.

(with the "FOOBAR\nBAZQUUX" part intangible, i.e. you cannot put the
cursor inside).

Now go to the 't' at the end of "BAZQUUXt" and type "C-x =".  In Emacs
28 and earlier you will see "column=28", in Emacs 29 you will see
"column=32".  This is because Emacs 28 ignores the display string and
examines only the buffer text, which is "covered" by the display
property.  By contrast, Emacs 29 ignores the "covered" buffer text
(since it's invisible on display) and examines only the characters of
the display string.  Since the newline has a width of zero, we get 9
columns between "file" and "t with" in Emacs 28, but 13 in Emacs 29.

So yes, Emacs 29 behaves differently, but the important part is that
both Emacs 28 and Emacs 29 are wrong: the _real_ column at 't' is 7
(columns are zero-based).  IOW, both Emacs 28 and Emacs 29 fail to
account for the fact that the newline inside the display string resets
the column to zero.

I could perhaps come up with a solution for that, and have the above
snippet correctly return 7 as the column.  But I don't see how this
will help company-mode and/or Eglot, since the value returned by Emacs
28 makes no more sense than the value returned by Emacs 29.

So I need your help with understanding what exactly fails in the
original use case, due to this change.  Specifically, how does the
code involved in this use move-to-column/current-column when there are
display strings with embedded newlines around?

TIA