Re: master 1e3b0f2: Improve doc strings of project.el

[Dmitry Gutov]

On 13.07.2020 07:40, Eli Zaretskii wrote:

> Sadly, you never answered my questions.  I still hope that you do,
> because I don't understand the staunch objections, to the degree that
> causes extremely unkind responses, to improve the doc string by saying
> something that is both simple and correct.

I have tried my best to be polite and accommodating in this thread, 
until you decided that your opinion is more important. And it's not the 
first time we're having this problem.

> Instead of answering my practical questions, which could have led us
> towards some acceptable compromise, you've elected to reiterate your
> views,

s/views/design goals/g

> which just makes another round of the same dispute we've been
> having, without any hope for reconciling the differences.  Because it
> turns out we disagree even more than I thought, on the very principles
> behind this, see below.

This sounds like you're saying that my opinion is invalid. I hope we 
manage to end up somewhere else than shouting "NO U".

> When the form of the return value changes, we will update the
> documentation.  This kind of thing happens all the time in Emacs
> development, and is nothing to be afraid of.

Having private/internal variables and private functions which we choose 
not to document or refer to in docstrings happens just as often.

> Especially if you
> consider some information "private", which means we under no
> obligation to preserve its form.  We even have an "Internals" chapter
> in the ELisp manual (woefully incomplete, but not because we decided
> not to tell those missing details).

If this stuff was really implemented in a way that is hard to make out 
for a would-be contributor (like, for example, the Display Engine is), 
then of course we should add a manual chapter with implementation details.

The information you want to document here, though, can be determined by 
anyone with basic Elisp skills in 10 seconds.

But in principle, yes, we could add a similar kind of "Internals" 
chapter to the manual that would outline how project.el works, and what 
is it for. If we decide that it's important enough.

And still, we wouldn't refer to that chapter from the docstring of 
'project-current'.

> This is why I'm frankly astonished by your opposition to document the
> return value.  Once again: what's the catastrophe in doing that?

Once again: it would encourage the uses that we don't want. And would do 
little to help the uses that we do want.

> > Here are things we want third-party code to be able to do:
> >
> > - Consume the public project APIs, which are currently limited to
> > 'project-current' and several generic functions defined for project
> > instance values, to define new commands or supplement existing ones
> > (e.g. to query the current project for its root or its list of files).
>
> Where is this documented?  I don't think I see it.

Orthogonal, but fair. I should copy some of these explanations to the 
package's Commentary.

> > - Create new project backends. Note that we don't want *everybody* to do
> > that because creating a well-performing backend is a significant amount
> > of work (unless the user has very particular needs and works with small
> > projects). But we want this to be a solid option, and it would be ideal
> > if, say, a handful of backends become popular someday. To create a
> > backend, one needs to choose the format of its instance value (which
> > shouldn't match any of the existing formats), write its
> > project-find-functions element, and define the project-root override
> > that dispatches on its instance value. Preferably, also implement
> > overrides for project-ignores and project-files.
>
> Where is it documented what needs to be done for creating a new
> backend?  Again, I don't think I see it documented.

Ditto.

> > Neither of these options requires knowing the exact format of
> > project-try-vc's return value, if only to stay away from it. But the
> > odds of a collision seem very low (and there are ways to reduce them
> > even further).
>
> These are just some of the possible development directions.  There are
> others, no less legitimate ones.  In particular, minor improvements to
> the code, which don't change the overall design and implementation.
> These and others do require to be aware of the form of the return
> value, or at least they can be helped by knowing that.

They shouldn't be documented, however, in the docstring of the main 
entry point to the Project API.

> Moreover, I'm quite sure that if you try to document the missing
> pieces mentioned above in a way that is useful for potential
> contributors, you will find soon enough that the policy of hiding
> useful information about interfaces, even "internal" ones, works
> against you: it makes it much harder to say something useful about how
> the various objects and methods interact or should interact.  You will
> eventually be forced to say "read the code" all the time.  Which means
> the documentation cannot do one of its important jobs: help to
> understand the code.

But I'm not having this problem.

What remains problematic, on the other hand, is referring to "project 
instances" in general, because it's a novel notion for Emacs Lisp API 
(having arbitrary values), and it doesn't invoke the same mental image 
that our more common terms do.

> But more generally, it is not our prerogative to decide for others in
> what directions they may or may not take the development in the
> future.  This is not our pet project, this is a community project, and
> any member of the community is welcome to propose changes to any part
> of Emacs.  If the change is useful for the users, and is done cleanly,
> we welcome and accept such changes -- this is how Emacs was and is
> being developed for 40 years.  It is what brought us where we are
> today.

That's a fairly general response to my fairly specific points.

> So I very much object to the policy you describe above, which is that
> you personally decide up front what kinds of changes are and aren't
> legitimate,

I challenge your authority to counter my design decisions. They weren't 
done in a day.

> based on some principles that are not necessarily shared
> by others (certainly aren't documented anywhere in the Emacs
> contribution-related docs), and then unilaterally implement this rigid
> policy, right down to the level of the smallest detail in the doc
> strings.  This is an extremely radical and harsh policy, which I don't
> think ever existed in Emacs, nor do I think it should exist.  If
> accepted, it will IMO harm the Emacs development in the long run.

We've had private variables and functions in Emacs for years. Probably 
more than a decade, I'm not sure.

> > Here are, on the other hand, things that people generally shouldn't do,
> > and yet I have seen them done:
> >
> > - Extract data from the project instance value directly, instead of
> > going through e.g. 'project-root'. For instance, writing (cdr
> > (project-current t)) instead of (project-root (project-current t)). The
> > former is bad code.
> >
> > - Write a new project-find-functions element which returns an instance
> > value that belongs to a built-in backend (usually, the 'transient' one).
> > AFAICT, this is usually done by cargo culting, without understanding of
> > the consequences.
>
> See above: I disagree very much with saying flatly that we don't want
> these.  These types of changes are not "verboten" in principle, they
> should be each one of them judged on its own right, and if done
> cleanly and efficiently, they are not in any way "wrong".

I invite you to read that again, first item in particular.

If you don't understand how it is a problem, you don't understand how 
this API works.

> In a Free Software project, you cannot forbid people from trying
> different development directions, even dangerous ones, you can only
> warn them about the dangers by documenting them.  Which is one more
> reason for having good, detailed documentation.  Refraining from
> mentioning something because you think it is dangerous is the wrong
> way, because people are curious and will find what you are trying to
> hide.  When they do, no amount of "I didn't want you to know that, so
> don't do that" will help, believe me.

I am physically unable to forbid them, and I'm not trying to.

But writing good documentation means mentioning relevant information, as 
well as avoiding littering the docs with irrelevant details.

> > I don't want to prohibit the last option outright, but we shouldn't
> > leave any impression that it's legitimate either. And most problems that
> > people try to solve this way (predominantly: customizing project root
> > markers, AFAIK) should be filed as bug reports/feature requests.
>
> As I already said several times, the right way of doing this without
> throwing the baby with the bathwater is to document the value and then
> tell this form is subject to change without notice.  It is a very
> simple solution that keeps both goals in our sights without losing any
> of them.

If this results in client code that only functions when the current 
project backend is project-vc (and only the current version), but fails 
with all others, would you consider this a success?

> > but it doesn't give you any guarantee: after all, we need to make
> > sure project-switch-to-buffer's behavior makes sense for different
> > backends, even ones we didn't write.
>
> Irrelevant: the information I added was about a single _existing_
> backend.  If you have proposals how to say something sensible and
> helpful about others, I'm all ears.

I have put some thorough descriptions in the grandparent email.

> > Instead, it seems we have to document that said value should uniquely
> > identify the returned project and be "stable": not change over time or
> > when fetched from different buffers belonging to the same project. That
> > also means that the value shouldn't contain any caches inside of it.
> >
> > And once we add these requirements, the answer to "which projects are
> > equal" translates to:
> >
> > - The backend is the same,
> > - The instances describe the same project, but the exact fashion depends
> > on the backend. As it would be anyway.
>
> Of course, none of this is described anywhere in project.el.  It isn't
> even clear what objects are part of the design, what are the
> requirements from each object, how to establish of some arbitrary Lisp
> object is "an instance of a project",

To draw an analogy to a long-established feature:

How do you determine that an arbitrary function is a "completion table"?

> > I'm not sure where you could put this particular bit of
> > information, where it would both be relevant and yet not obvious from
> > the nearby code, though.
>
> _Any_ place is better than _no_ place.  "Not sure where to put" is not
> the same as "let's not document this".

Right. That's why I chose these particular words. But let's not put that 
in the project-current's docstring.