Re: progmodes/project.el and search paths

[Eric Ludlam]

Hi Dmitry,

Thanks for the thoughtful reply.

On 08/11/2015 04:01 PM, Dmitry Gutov wrote:
> Hi Eric,
>
> Sorry for the late reply. I wasn't really sure what kind of response 
> is expected.
>
> In short, I like your vision as a whole, but whether each particular 
> element of it can apply to project.el, should be investigated separately.
>
> On some of them:
>
> * I wonder what's the main use of having hierarchical projects vs. 
> simply tracking several currently open projects (or project roots). 
> And how the consumers of the API would take advantage of that 
> distinction.

I doubt consumers of the project API would care.  Nested projects is 
simply something the detection system should enable.  If there is a 
centralized way of scanning for projects, it would need to support 
this.  If each project system detects projects its own way, then it 
wouldn't matter at the core level.

My hypothesis is that project detection is tricky, and new custom 
projects would pop up more frequently if there were infrastructure for 
detecting them.  If project.el were to provide that infrastructure, it 
would need to support nested projects.

> * Project services seem to be the proper way to handle optional traits 
> of significant size that some projects have, and others don't.

> * "Project Loading safety" are probably out of scope. I'd expect the 
> majority of implementations to be based around loading a project build 
> (or settings) file of pre-existing format. The popular IDEs don't ask 
> the user whether they want to load the project file. On the other 
> hand, some project types might need to be enabled/visited explicitly.

Every project is indeed different.  A system that needs to load lisp 
code needs to protect against ill formatted lisp code.  Putting this in 
a core system will help implementations from all creating the same 
latent bug.  If project.el isn't going to provide detection services, 
then it won't need this feature.

> * WRT "Detection ordering", the current project-find-functions 
> structure might be enough. Any Makefile project implementation will 
> most likely go before the VC project anyway (unless the respective 
> add-hook form is told to APPEND, for some reason). And it's hard for 
> us to expect for third-party implementations to be able to declare 
> priorities between themselves.

Project.el is providing a list of ways for projects to present 
themselves.  At the moment however, projectile v EDE v your vc detector 
could all claim ownership of the same project.  At the moment it 
probably doesn't matter because since all three provide different 
services based on their personal consideration of 'root'. Any tool 
needing to know only the root would be happy, but if there were some 
extra service to be provided, a priority between the 3 would be needed.  
If project.el were to provide the detection service as well, then the 3 
would need to prioritize.

In EDE, a project adds itself to the list of possible projects. Each 
project classifies itself by a term that places itself in one of three 
bins, and hopefully those bins don't conflict. ;)

Anyway, EDE's bins are:

'unique - something like the Emacs or Linux project - detect these first.

 no flag provide - One of the specialized build system projects like 
Automake support, or Maven.

'generic - Something that provides very little overall support, such as 
detection by VC, or existence of SCons. - Always at the end.

This is pretty easy to self bin by.

> * "Update meta-data" might be handled by each respective 
> implementation, in conjunction with auto-revert-mode.

Right.  Each project implementation needs to update their data from the 
file system their own way.  If there are generic development menus, it 
would be helpful for the user to have one way to update project meta 
data regardless of which type of implementation is active.

> * "Project browsing" shouldn't need much more than project.el exposes 
> already.
Yup.  I was trying to be complete.

> * "Project Local Variables" look very similar to project metadata.

They are a type of meta data that let users provide extended behavior to 
the project.

> * The "'include path' concept" looks very similar to 
> project-search-path. This might be the main purpose for the 
> project-search-path entries that lie within project-roots.

I will need to look at your search path concept more.  Are there 
multiple paths for different purposes?

> * WRT cscope, etc, integration, I think the main unsolved problem we 
> currently have is that they aren't re-run automatically. Nor do we 
> have a command that would do that for the user on demand.

Several of these tools have an 'update the current tags file' type mode 
you can get at.  I don't recall if I wrapped that up with a handy 
function or not.

> * "Load a project" seems incompatible with the project-find-functions 
> approach. Which is unfortunate.
>
> * "navigating between code and doc", as stated, doesn't seem 
> particularly useful: if you're reading the source code of a function, 
> the docstring is usually right above. Jumping from the doc to the 
> source is better, but this is closer to the domain of xref.

Just pulling out examples, some were from threads on the ML.

The key concept is that a project is rarely just one single activity.  
Usually there are doc activities, code activities, build system 
activities.  Some projects have Java activities, and native C++ 
activities.  All these different things are for a common goal, and often 
there is a common lingo between them.  I represent that lingo as 'tags' 
in Semantic, and it enables you to 'copy' a tag which includes the meta 
data around the name and 'paste' it into a buffer of a different 
language.  The only useful one there can turn a lisp tag into a texi 
@defun block, but you could imagine a way to update your java native 
methods bidirectionally.   The project knows how to link these systems 
together, such as finding the right doc for your Lisp code.

This is an advanced maneuver that would be great for a tool author to be 
able to do, and is more likely if that kind of association can be easily 
made.

> * "Language specificity" might be useful, but in practical terms it 
> sounds like adding an optional (?) "language" argument to 
> project-search-path and any similar method that we add later. Which 
> complicates implementations. Or just have an Emacs-global repository 
> of file name patterns corresponding to languages? Not sure.

Language specificity was key for me because of the way mode-local works, 
and how each language had customized behavior for the parsing tools.  I 
tended to focus on language specific accurate behaviors, and not generic 
potentially inaccurate features.  For example, smart completion vs dabbrevs.

As noted somewhere, symref had a way of converting modes to patterns.  I 
had to ask around, but eventually got something pretty good.

[...]
> > I consider this file incomplete, but I hope it proves useful as a
> > starting point.  I'd be happy to periodically update it for a little
> > while if it is useful.
>
> Were we using a nice issue tracker, I'd have asked to create a 
> tracking issue (or would've created one myself), but in the present 
> situation, I guess you'll have to resend it with updates.
>
> But in any case, I'd say technical proposals on specific features, as 
> well as patches, would provide more value here, rather than 
> maintaining a high-level overview of things we might get around to someday

My goal is to be more helpful than adversarial, but at the time I didn't 
have a clear picture where project.el would fit, and if there were 
anything I had to deal with in the future.

I'd be glad if I could stop maintaining parts of EDE because it became 
part of some core system.   It is still unclear to me where the 
boundaries lie.  I'm not sure where a patch would be helpful yet.  My 
hope is that my document would help you define the bounds of project.el 
based in a set of terms we can both agree to.  It would then be obvious 
how I could help by donating pieces of EDE as needed.

My recommendation for you is to perhaps not reply to individual 
clarifications above, but perhaps define where you think project.el fits 
into the whole using whatever you like from my doc, and that would help 
us focus the conversation on the boundary, and not on random things I 
invented that aren't close to the boundary.

Thanks
Eric