bug#54079: 29.0.50; Method dispatching eratically fails

[Stefan Monnier]

Alan Mackenzie [2022-03-09 20:32:59] wrote:
> On Wed, Mar 09, 2022 at 13:06:11 -0500, Stefan Monnier wrote:
>> >> I don't understand the scenario you're thinking of.
>> >> Are you thinking of something like `(eval-when-compile (byte-compile 
>> >> ...))?
>> > Yes.
>> >> Does that ever happen in real life?
>> > Probably exceedingly seldomly.
>> > What's to be gained by not catering to this unusual case?  What do we
>> > lose?
>
>> We lose making it work right for the 99% other cases that *do* occur?
> How would it not work right for such a case?  Can you give an example?

I'm not sure what "such a case" you're thinking of.  But in general,
evaluation of code doesn't expect symbols to have positions: it may test
`eq` between symbols and it may be run without having the magical
`symbols-with-pos-enabled`.

So as a general rule we should strip symbols before we pass it to `eval`.

>> >> >> And why bother stripping the result of `byte-compile-eval`?
>> >> > Because it might be the result of evaluating a defun (or defvar or
>> >> > defconst).
>
>> >> AFAIK sympos should only appear within the compiler pipeline between the
>> >> "read" and the "emit resulting bytecode".  They may be passed to various
>> >> functions and macros along the way, but I can't think of any scenario
>> >> where they'd end up returned by `(byte-compile-)eval`.
>
>> >> > This was the situation which gave rise to the bug.
>
>> >> Could you give some details about how it played out?
>> >> [ Either here or as a comment in the code.  ]
>
>> > Michael byte compiled cl-generic.el.  This created cl-generic.elc
>> > correctly, but also left uncompiled forms in the function cells of the
>> > symbols defun'd inside an eval-{when,and}-compile.  These forms
>> > contained symbols with positions.
>
>> Hmm... we're talking about stripping the result of `byte-compile-eval`.
>> This function is only used for `eval-when-compile`, not `eval-and-compile`.
>> And nothing in your above description indicates that the sympos appeared
>> in the resulting value of `eval-when-compile` (as opposed to appearing
>> in the slot of functions and variables that were set during the course
>> of the evaluation).
>
> OK, sorry, I was mistaken.  These forms with SWPs arose from
> evan-AND-compile, which doesn't use byte-compile-eval.

OK, now can we get back to the question:

    And why bother stripping the result of `byte-compile-eval`?

>> >> >> Fundamentally, `eval` should always strip before doing its job.
>> >> > Except when what it's evaluating is a defun, defmacrro, defsubst, etc.
>> >> Why?
>> > Because that evaluated form might later be byte compiled, and the SWPs
>> > will be needed for that.
>
>> I don't understand the scenario you're thinking of.
>> Are thinking of a case like:
>
>> - something causes the execution of (eval '(defun foo ...))
>> - the user types `M-x byte-compile RET foo RET`
>
> Sorry again, I've lost the thread here.  Weren't we talking about
> eval-{when,and}-compile, not eval?

No, the text you cited even included my original statement:

    Fundamentally, `eval` should always strip before doing its job.

> Inside these two special forms, we should preserve the SWPs as long as
> possible (i.e. as long as they won't cause any errors).

We should preserve them while macroexpanding and compiling their
contents, yes, but then we should strip the symbols before we pass the
result to `eval`.

>> If so, then:
>> - I don't think we should care about this case because it's extremely
>>   rare and fundamentally broken (the symbol's function cell contains
>>   a function *value* (i.e. a closure) and not a function's source code,
>>   so in general we need `byte-compile--reify-function` which implements
>>   a heuristic to go back to something like a source form, which can
>>   break in various ways in corner cases).
>
> Really?  After evaluating a defun, etc., we have a lisp form in the
> function cell, which may be a closure.

A closure is not "a Lisp form".  In general passing a closure to `eval`
may signal an error because it may very well be an invalid form.

The body of a closure is a Lips form, yes.  But that's not what's inside
a symbol's function cell.

> The function byte-compile compiles an arbitrary form, doesn't it?

Try the following:

    ;; -*- lexical-binding: t -*-

    (let ((x 0))
      (defun my-inc1 ()
        (interactive)
        (message "x=%S" (setq x (1+ x))))
      (defun my-inc2 ()
        (interactive)
        (message "x=%S" (setq x (1+ x)))))

load that file.  Then try `M-x my-inc1` and `M-x my-inc2` and you'll see
that they correctly increment the same counter.
Now do `M-: (byte-compile 'my-inc1)`.
And try `M-x my-inc1` and `M-x my-inc2` and you'll see that suddenly
they each have their own counter.

That's because it's one of the corner cases that
`byte-compile--reify-function` can't handle correctly.

>> - If we don't strip before calling the `M-x byte-compile` then the code
>>   may/will bisbehave because of the presence of the sympos.
> How?  byte-compile is designed to use SWPs.

The misbehavior I'm referring to is what happens when you call the
function before you byte-compile it (or, more likely, when you never end
up byte-compiling it), because the presence of sympos in the function
will mess up its semantics (because `symbols-with-pos-enabled` won't be
set any more when it's called).


        Stefan