bug#44653: 28.0.50; sql-mode gets confused about string literals

[Dale Sedivec]

On Jan 26, 2021, at 21:57, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> Dale Sedivec <dale@codefu.org> writes:
> 
>> I think `syntax-ppss' has started returning incorrect information about
>> apostrophe-delimited strings in sql-mode in master. [...]
>> 
>> Steps to reproduce:
>> 
>> 1. emacs -Q
>> 
>> 2. Evaluate the following in *scratch*:
>> 
>>      (let ((buf (generate-new-buffer "sql")))
>>        (switch-to-buffer buf)
>>        (sql-mode)
>>        (insert "select '''")
>>        (goto-char 1)
>>        (delete-region 1 8)
>>        (goto-char (point-max)))
>> 
>>   Point should now be at the end of an `sql-mode' buffer containing
>>   "'''" (three apostrophes).
>> 
>> 4. Press backspace to erase the third apostrophe.
>> 
>> 5. M-: (nth 3 (syntax-ppss)) RET
>> 
>> Expected result: fourth element of syntax-ppss, the delimiter character
>> for the current string, is nil, since we are no longer in a string
>> 
>> Observed result: fourth element is ?' (39), indicating that point is
>> still inside a string
> 
> I can reproduce this behaviour...  but if I then type, say, "a DEL",
> then (nth 3 (syntax-ppss)) returns nil.
> 
> So it seems like syntax-ppss doesn't recompute the status until a new
> character is inserted...  which I think makes sense?  Until you've typed
> something more, Emacs doesn't really know whether we've entered a new
> syntax state or not here.

Thanks for looking at this.

Do I correctly understand your statement to mean that parse state is only 
updated when characters are added to a buffer, not when characters are deleted? 
 If so, that would indeed seem to mean that this is not a bug.  I tested 
something similar in a python-mode buffer, and indeed syntax-ppss still thinks 
it's in a string when I delete an apostrophe, just as in the above example.  
It's possible I just missed that explanation in the manual.

If things are working as expected, please feel free to close this.  It's 
probably inconvenient that syntax parsing can't be relied upon after deleting 
characters, but I can imagine that changing this is not simple.

> (I'm also wondering what the actual bug you're experiencing is, since
> I', guessing you don't go typing M-: (nth 3 (syntax-ppss)) RET at random
> just for fun.  :-))
[...]

While acknowledging that I sometimes forget the names of my family members: I 
*think* I was trying to change how apostrophe (') behaves in sql-mode buffers 
with electric-pair-mode turned on.  In case it helps, or you're just curious, 
specifics follow.

#40231 (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40231) made a change to 
the syntax of apostrophe in an SQL string, which made electric-pair-mode behave 
in a way I did not appreciate.  While writing code to make ' behave in the way 
I expected, I ran into difficulties due to the behavior of syntax-ppss I 
described in my original message.

To demonstrate what I was trying to change:

1. Create an sql-mode buffer with electric-pair-mode turned on

2. Type ', which results in '|' with point at |

3. Type ' again

Desired behavior, and pre-40231's patch behavior (IIRC): ''| with point at |

Post-40231 behavior: ''|'

I wanted the pre-40231 behavior.

Here's what I'm currently using to achieve the pre-40231 behavior (it ain't 
pretty and I'm struggling to remember why the third cond clause is there):

;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40231
;;
;; In addition to test case in comment below, also try '''''' at BOB
;; and try inserting ' at BOB with '' in front of point.  Also make
;; sure apostrophes don't pair in comments.

(defun my:sql-mode-electric-apostrophe (arg)
  (interactive "P")
  (cond
    ((or (not electric-pair-mode)
         arg)
     (call-interactively #'self-insert-command))
    ((and (eq (char-after) ?')
          (eq (nth 3 (syntax-ppss)) ?'))
     ;; We were already at a string, and the character after point is
     ;; an apostrophe.  Just move beyond it.  (Behavior changed after
     ;; changes from Emacs bug #40231.)
     (forward-char 1))
    ;; XXX I have no idea if any of this makes sense if/when
    ;; https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44653 gets fixed.
    ;; Check back then.  I think `self-insert-command' should be
    ;; sufficient, but ISTR I had to avoid using it in this case
    ;; because elec-pair was confounding me?  I don't quite remember.
    ((and (eq (char-before) ?')
          (not (nth 3 (progn
                        ;; (syntax-ppss-flush-cache (- (point) 2))
                        (syntax-ppss)))))
     ;; (self-insert-command 1)
     (insert "'")
     (save-excursion (electric-pair--insert ?')))
    (t
     (self-insert-command 1))))

(with-eval-after-load 'sql
  (define-key sql-mode-map "'" #'my:sql-mode-electric-apostrophe))

Dale