bug#18158: D Mode: Getting rid of the ugly advice on looking-at.

[Alan Mackenzie]

Hello, Vladimir,

Thanks for your extensive reply.  I have been thinking it over over the
past few days.

On Sun, Feb 02, 2020 at 16:59:43 +0000, Vladimir Panteleev wrote:
> Hello Alan,

> First, thank you very much for reaching out. I held the expectation
> that the maintainer(s) of cc-mode would not have any time to spare on
> derivative modes distributed outside of Emacs, so your message was a
> welcome surprise.

:-)

> I greatly appreciate you taking the time to prepare a patch for
> d-mode. However, I hope you will forgive me if I burden you with a
> bigger matter.

No problem.

> As of last year, I had been working on a major update to d-mode. This
> new version fixes all known bugs in d-mode filed so far and has many
> other improvements. It is located here:

> GitHub pull request:
> https://github.com/Emacs-D-Mode-Maintainers/Emacs-D-Mode/pull/93
> Git clone URL: https://github.com/CyberShadow/Emacs-D-Mode.git ("next" branch)
> Current version:
> https://raw.githubusercontent.com/CyberShadow/Emacs-D-Mode/next/d-mode.el

> I'll be happy to create non-GitHub mirrors of the above upon request.

I think I can get a copy of the repository without having to sign up,
but I haven't had a chance to do this, yet.

> Unfortunately, the only way I found to achieve these improvements in D
> language support was through advising even more cc-mode functions,
> similar to the "looking-at" advice your message pertains to.

Maybe we could work together on this so that these pieces of advice
could be minimised, or even avoided altogether.

Could you possibly give me one or two examples of why these pieces of
advice are needed?  This would give me more of a feel for what the
problems are.

> I fully realize that this approach is not very future-proof. I tried
> to keep the implementation as unintrusive and unreliant on cc-mode
> internals as possible. However, cc-mode is a non-trivially large and
> complicated piece of software, and D is a relatively large and
> complicated language, so I admit I have likely fallen short of that
> goal.

CC Mode is indeed (too) complicated.  Partly this is because the
languages it deals with are complicated (particularly C++), but also, it
has to be said, because the maintainers haven't always adopted an
optimal compromise between speed, accuracy and simplicity.  Accuracy has
tended to be the highest goal, followed by speed.

> My plan for ongoing maintenance was to keep up and address problems
> due to changes in cc-mode internals as they occur; however,
> considering the amount of time required to debug and devise fixes for
> each problem, I may have been overly eager in considering this
> approach.  For this reason I had been thinking of contacting you for
> advice on how best to proceed.

> Some possible ideas I had in mind:

> 1. Add hooks and variables to cc-mode to allow d-mode to cleanly
> configure cc-mode to support the D language.

This would perhaps be the cleanest option.

>    Though I believe D firmly fits into the Algol-like family of
> languages currently supported by cc-mode, it also has many unique
> features, so I worry that this approach would be overly burdensome to
> cc-mode.

How does D compare with C++ in terms of complexity?

One practical step for option 1 would be for you to have write access to
the CC Mode repository, in particular to a "D Mode" branch, where you
could implement the necessary CC Mode features to support D Mode.
Something like this was done ~10 years ago for enhancing Java Mode.

> 2. Move d-mode into cc-mode, replacing advice hacks with clean integration.

I would be less keen on this.  Some while ago, my predecessor Martin
Stjernholm decided as a matter of policy not to have any more languages
directly implemented in CC Mode, and I have continued with this.  The
problem is that if the maintainer of a particular CC Mode mode stops
contributing for any reason, the mode becomes old and fossilised.  This
has already happened to IDL Mode and Pike Mode.

But perhaps a D Mode support file could be included in CC Mode, somewhat
along the lines of how cc-awk.el supports AWK, but including only
essential support for D Mode rather than including D Mode itself.  I'm
not sure if I've explained that very well.

>    I would be happy to contribute my work to GNU and continue
> supporting d-mode within cc-mode, if this is a possibility. However,
> even though most of the code in my "next" branch was written by me,
> the project has been around for a while, and I don't know if we can
> find all authors and have them transfer copyright to FSF; or,
> alternatively, move only the code I've written, and rewrite the
> missing parts, but I don't know how strict we would need to be in this
> regard.

Very strict indeed.  The FSF's policy is that all contributions which
aren't minor (in practice, up to about 15 lines of code) need their
authors to assign their copyright.  I've not known them to deviate from
this policy, which has good reasons, though occasionally it can cause
bad feelings.

> 3. Reimplement d-mode in cc-mode.

>    It might be easier to start from scratch and, using just the
> existing d-mode test suite for reference, reimplement d-mode within
> cc-mode. Not having to resort to hacks such as advising functions will
> likely allow a much cleaner implementation than the current one.

Restarting an existing piece of software from scratch is a risky
process.  One skirts existing bugs cleanly, but there's a high chance of
introducing new bugs which weren't there in the first place.

> 4. Copy relevant parts of cc-mode into d-mode.

>    One very ugly, but possibly practical approach would be to include
> all relevant cc-mode code in d-mode's distribution. All definitions
> would need to be properly namespaced to not conflict with the real
> cc-mode. This would effectively "freeze" the cc-mode version that
> d-mode uses, thus protecting it from incompatible changes in internals
> of future cc-mode versions, though compatibility fixes for future
> Emacs versions would still need to be applied, and d-mode will be
> unable to benefit from any future cc-mode improvements.

I don't think this is the best way forward.  Even though it could work,
it's likely to involve a lot of unrewarding work, and some duplication
of effort in fixing bugs common to CC Mode and D Mode.

> 5. Abandon ship, and scale back D language support in d-mode to a
> maintainable minimum.

This would be entirely up to you, but I suspect that D Mode users would
be unhappy with losing useful features for the sake of maintainability.

>    Perhaps using cc-mode for D language support within Emacs is not an
> optimal solution. All of the above ideas would certainly require a
> non-trivial amount of further effort to achieve, and the result would
> only cover the combination of supporting the D language within Emacs.
> An approach that has been recently gaining popularity is, instead of
> writing one package for each language for each editor, to use
> "language servers" which use a common protocol to provide language
> support to any editor. The LSP protocol specification (as maintained
> by Microsoft) supports formatting (indentation), outlining (imenu),
> and (in recent versions) semantic syntax highlighting (fontification),
> with the downside of requiring additional software and configuration.
> If there is no interest in supporting D / d-mode in cc-mode (ideas 1-3
> above), then perhaps this may be the best course of action.

This is a big topic indeed.  There have been a few discussions around
LSP on the Emacs developer mailing list in the past few months.  Using
LSP in place of the current mechanisms would involve a lot of work, but
would likely be worth it.  Recently, somebody has implemented
fontification for Java Mode using Emacs's own analysis suite (whose name
I can't recall at the moment).  So it seems things are moving in that
direction.

Personally, I have mixed feelings about LSP.  It may be a better way of
doing things, but comes along at a time when traditional CC Mode is very
highly developed over several decades of hard work.  It's a bit like
when vinyl records were superseded by CDs just around the time when we'd
learnt how to make high quality records.  For a while, the new CDs
weren't as good.

> I would love to hear your thoughts on the above.

I hope I've given you enought to think about.  ;-)

> Also, looking at the email headers, I see that this message was in
> reference to Emacs bug 18158. Looking at the previous messages in the
> bug thread (https://debbugs.gnu.org/18158), I found it odd that a bug
> in d-mode's behavior was filed at Emacs' bug tracker.

It's bound to happen every now and then.  I actually missed this bug
back in 2014 and wasn't aware of it until Stefan Kangas, who's been
going through hundreds of open old bugs trying to settle them, alerted
me to it.

> I think that any d-mode or cc-mode issues observed when d-mode is
> active ought to be reported to the d-mode bugtracker, and not the
> cc-mode one. It is certainly not my intention to burden cc-mode
> maintainers with d-mode's bugs or shortcomings. Perhaps there is
> something I need to do to make it clearer where to report bugs in
> d-mode?

I don't think you need to worry too much, very few D Mode bugs get sent
to the CC Mode or the Emacs mailing lists.

> In any case, I've been unable to reproduce the bug (as it was
> originally filed, and with the test code provided in message #20
> (<8B3ABB9B-7A39-428A-A65B-B77D0614C9FE@gmail.com>), on either the
> current master branch or my "next" branch of d-mode. I see that the
> bug was filed in 2014. Perhaps it was fixed by this commit in 2015:
> https://github.com/Emacs-D-Mode-Maintainers/Emacs-D-Mode/commit/27fbe66f6de27f8337fe40d6a19f039c589cd1fc

> I added a test case for this fix in this commit:
> https://github.com/Emacs-D-Mode-Maintainers/Emacs-D-Mode/commit/3e5a5d523d7580b2a42f376cbf83bdd9ac296bbd

To reproduce it, I temporarily removed the advice on `looking-at' from D
Mode.  The problem was then apparent in the test code supplied by Liran
Zvibel on 28th January to me, Cc: to the Emacs bug list.

> Best regards.

-- 
Alan Mackenzie (Nuremberg, Germany).