bug#55338: Symbols with position are sometimes not symbols

[Alan Mackenzie]

Hello, Paul.

On Mon, May 09, 2022 at 10:25:42 -0700, Paul Eggert wrote:
> While looking into another issue I saw the code added a few months ago 
> for symbols with position, and noticed what looked to me to be a bug: 
> although (read-positioning-symbols "xyz") returns a symbol, (symbolp 
> (read-positioning-symbols "xyz")) returns nil.

Yes.  A symbol with position isn't a symbol, any more than a symbol is a
function.  But (functionp 'list) returns t, nevertheless.  This doesn't
appear to give rise to confusion (except, perhaps, in novices).

> It seems that symbols with positions are symbols only if 
> symbols-with-pos-enabled is t; otherwise, they are not symbols. 

SWPs are not symbols.  They merely give a result of t to symbolp.

> Unfortunately this leads to confusing behavior such as the above.

Why is this any more confusing than functionp returning t for a symbol?

Perhaps the documentation for SWP could be firmed up a bit.

> As I understand it, the symbols-with-pos-enabled variable is present for 
> performance reasons only. If this understanding is correct, I suggest 
> removing the variable and having symbols with positions always be 
> symbols.

This is an interesting idea.  It would increase the amount of space used
by Emacs's symbols, though.  Would this be important?

> This would avoid the confusion. I think this could be done without
> hurting CPU performance compared to the master branch when
> symbols-with-pos-enabled is nil; see attached patch, which is relative
> to master commit 6fc54786c3bb797068675d7eb7b500fb990bd04a. (The patch
> is incomplete, as src/comp.c would need to be updated to match, but
> that's merely turning the crank.)

So in place of checking symbols-with-pos-enabled at each otherwise
failed EQ operation, we need to check the "real symbol" of each symbol
in such a comparison.

How much faster or slower is, say, a bootstrap build (even without
native compilation) with this proposed scheme?  What about benchmarks
which don't involve compilation?

> Even if this patch is completed I have qualms about performance and 
> correctness of symbols with positions.

The performance appears to be OK in tests of real world usage, with
perhaps a maximum 1% to 2% slowdown observed.  Benchmarking some test
programs showed a greater slowdown.  More benchmark results would be
welcome.

As for correctness, the code has been running for several months with
just one or two pertinent bugs found in the byte compiler.  They can be,
and have been, corrected.  Critically, the bug which gave rise to the
mechanism, wrong line/column numbers being reported in compilation
warning messages, has been fixed.

> How about a new build-time flag --disable-symbols-with-pos that would
> disable the feature, for people who don't want performance degraded
> compared to Emacs 28, or who are unsure of the correctness
> implications of the new feature?

Symbols with position is NOT a user feature.  It is (an essential part
of) a bug fix.  It seems inappropriate to offer an option not to fix a
bug.

> That should be easy to add, and I can look into adding it unless
> there's objection.

I object.  I doubt it would be easy to add, since you'd have to
resuscitate a fair amount of old code discarded from master some while
ago.  Maintaining two versions side by side would lead to maintenance
complications.

> Also, there should be a NEWS item about symbols with positions.

No.  Again, symbols with position is not a feature, and is not intended
for use outside of the byte compiler.  Although I suppose it could be
used if anybody wrote some compiler-like feature for Lisp code, but such
a hacker will know about SWPs without needing to read about them in
NEWS.

[ Patch snipped, but perused. ]

-- 
Alan Mackenzie (Nuremberg, Germany).