Re: Requesting for review of the Draft proposal for - procfs

[olafBuddenhagen]

Hi,

On Wed, Mar 26, 2008 at 09:21:08PM +0530, Madhusudan C.S wrote:

>   I have come up with this draft proposal so that we can discuss
>   further based on the this proposal.

It looks very promising already :-)

> Please review it and suggest any kind of mistakes including spelling
> and grammatical errors.

Don't worry about that -- we won't judge your proposal on spelling or
grammar :-)

>      I am planning to make as many revisions as possible(atleast 10)
>      to this proposal. But I do understand that any number of reviews
>      and revisions will not be sufficient though I am running out of
>      time and have to submit it as soon as possible. I am planning to
>      submit it through Google Application Applet on Thursday hoping
>      that all the reviews and revisions will be over by then.

I think you should submit it now, so we have a better overview of all
proposals. You can still improve your application afterwards! :-)

>    Another important note is that, this document is the detailed
>    proposal I have written. I have put all the ideas in my mind
>    together so as to make it clear to you people about what I am
>    thinking. This proposal is nearly 13,000 characters and in no way I
>    can submit this through Google Applet since it restricts the lenght
>    to 7,500 characters. My final submission must be a heavily tailored
>    version of this document. I have thought of cutting down the
>    description of each of the core functionalities I have written in
>    project details part and also the details of the schedule, since I
>    will be making the full version of this document available through
>    my blog. I request you people too to suggest me which other parts
>    require tailoring.

You could just refer to an external source for most of the "project
details" section, I'd say...

Now to your actual proposal. (BTW, as a general note: If you want people
to read stuff you post to a mailing list, better include the contents
right in the mail body, rather than linking to it. This lowers the
barrier to reading it, and more people will do so...)

> [...] So the idea now is to do a rework on the existing procfs code to
> make it more useful, reliable, robust and thus easily extendible along
> with adding a huge number of functionalities to procfs itself that are
> more compatible with GNU/Linux. These additions will make procfs
> highly useful for implementing process management tools like ps, top
> etc. easy.

This already hints at something that seems to be a misunderstanding,
which continues throughout the rest of your proposal: The idea of this
project is *not* creating a procfs that is similar to that on Linux, and
implementing other tools on top of that. Rather, the idea is to
implement a procfs that is fully compatible with the Linux one (for the
parts that it actually does implement), so that the existing GNU/Linux
tools using procfs can be used out of the box!

> The project begins with migrating the existing procfs code which uses
> libtrivfs, to use libnetfs through out procfs. The necessity is that
> libtrivfs is a library which is best suitable for filesystems
> providing a single directory or file. Since procfs is now implemented
> as a virtual filesystem with a lot of files, directories and
> functionalities a more reliable and robust library is required. So
> libnetfs provides best answers to these needs since it provides
> functions for both network filesystems and large virtual filesystems.

This is a good explanation :-)

> The next stage of the project is to develop a standard set of
> functions(I will call this as Intra-procfs Programming
> Interface(IpPI)) which provides basic functionalities required to
> setup a virtual filesystem, in particular procfs. These are nothing
> but the callback functions that use the services of the libnetfs
> library but provides the interface to implement procfs in particular
> (Advantages in Benefits section).

I'm not sure such a middle layer is really necessary. Don't
over-engineer. Start by implementing the actual functionality, and only
if you indeed encounter a lot of code duplication, think about improving
the infrastructure... Think KISS and YAGNI :-)

> The functionalities that form the immediate requirements for
> implementing process management tools like ps, top etc and hence those
> I intend to complete in the duration of GSoC are listed below along
> with a brief description on each of them:-
[...]

See the other mail for comments on those... (Sorry for the late reply.)

> One can now think of procfs problem comparable to networks problem in
> which monolithic designs were replaced by layered designs through OSI
> reference model and more practical TCP/IP architecture resulting in a
> robust design of networks. Similarly by layering the design of procfs
> which includes 3 layers 1.Intra-procfs PI 2.The core functionalities
> of procfs 3.procfs API We are making procfs system highly robust,
> reliable. So if new functionalities are to be added, its just a matter
> of using the IpPI. Also if some functions are to be changed its only a
> matter of changing the implementation of the function without
> affecting the other two layers and this applies to those two layers
> too.
> 
> The above discussed benefit is more in compliance with the highly
> modular approach of Microkernels and I feel this falls in line with
> the Hurd philosophy.

I understand your goal of achieving modularity; but that's not the right
way of doing it. Modularity in the Hurd is not achieved by sophisticated
layering within one process. Rather, it is achieved by splitting
functionality into individual servers (translators).

As Frederik already pointed out, one thing that we could do is splitting
out the global bits, like /proc/uptime etc. into individual translators
-- they are pretty independant, and there is really no good reason to
implement them all in one translator. Each of these files could be
easily provided by a simple trivfs-based single-node translator.

The per-process information, which form the core of procfs, are more
tricky. Ideally, we could use a multiplexer, which launches individual
translators for each pid and possibly also for each piece of information
per pid. However, we have very little experience with doing such things
efficiently. This is definitely outside the scope of this project.

While it might be interesting to look into such possibilities, should
you be able to finish the main functionality before time, I suggest to
completely leave it out of the plan, and stick to the existing
"monolithic" design for now... Getting a functional and compatible
procfs is presently much more important than experimenting with more
elegant design choices :-)

> This makes the life of Hurd developers easy, since it is now very
> simple to write process management tools.

Here the aformentioned misunderstanding shows again: The idea is *not*
to make it simple to write new process management tools, but to make it
possible to reuse the existing GNU/Linux ones...

> Deliverables
> 
> 1.A set of functions necessary to implement procfs filesystem.

I wouldn't consider this a deliverable on it's own. This is an issue of
internal code organisation. What really matters in the end is what the
translator is able to do, not what internal functions it uses for
that...

> 2.The core procfs filesystem itself and the above mentioned features.

Right, that's the goal of the project.

> 3.A set of functions that forms the API to procfs filesystem.

Not sure what kind of API you mean here. This might be a manifestation
of the misundestanding again?...

> 4.Installation files and scripts to use the services of procfs
> filesystem.

There should not be much required here. Basically we just need a
Makefile to build the translator; the rest is up to the system
distribution...

> A sample code which implements one of the process management tools in
> the simplest possible way to make things clear and to serve as an
> entry point to the tools developers. 

Misunderstanding again I'd say...

> Project Schedule
> 
> The project is planned to be completed in 7 major phases. Every phase
> includes documenting the progress during that phase, so its not
> mentioned anywhere specifically. The timeline and duration of each of
> these phases is given below:
> 
> 1.Initial preparation and migration (Community Bonding Period: has
> already started - May 25th) In this phase I will gain hands on
> experience on translator programming, and become well versed in it. I
> will also go through the Hurd code in general trying to understand its
> architecture in depth and will read as many documentations as
> possible. This phase also involves an initial rough design and
> migration of the existing procfs to libnetfs.
> 
> 2.Analysis and Design (May 26th - June 5th ) This phase involves the
> analysis of the previous migration and a complete design of the
> proposed /proc filesystem and also what needs to be done in next 3
> months in detail.
> 
> 3.Coding Stage I (June 6th - June 15th ) This phase involves coding of
> layer I, i.e IpPI.
> 
> 4.Coding Stage II (June 16th - July 20th ) Since this is the heart of
> this project and forms the crux as of now, this phase requires a huge
> lot of time. This encompasses the mid-term evaluation period and by
> this time features upto point 14 in the project details will be
> finished. Rest in the next week.
> 
> 5.Coding Stage III (July 21st - July 30th ) This phase involves coding
> of the last layer in the project, i.e procfs API layer.
> 
> 6.Testing, Requesting for community wide Reviews and evaluation (July
> 31st - August 3rd ) In phase I will closely interact with the entire
> community to help me in testing, reviewing the project's working, and
> I will request every willing developer to give his insights on the
> project and make the changes as per their suggestions. Also involves
> consolidating the Documentation.
> 
> 7.Packaging and Wrap-up (August 4th -  August 18th ) This final phase
> involves final testing and fixing remaining bugs if any. Working with
> the community to merge the project with the CVS HEAD of Hurd.
> Requesting users for Beta testing and wrapping up the documentation.

Your plan looks pretty sound all in all; but I have some doubts about
the middle phases (3-5): You want to work layer by layer. This is
problematic, because if the project turns out harder than expected, and
you can't complete it all, we are left with no visible improvement at
all...

I think it makes much more sense to implement/fix the individual bits of
information one by one. (In fact, you could try to implement one of the
easiest bits even now during the application phase, to convince us of
your programming skills :-) )

I guess it's our fault for not making it clear in the application
template: It's very important that the project yields visible
improvements, even if it can't be completed as planned for some
reason...

This way you can also better deal with any design problems you encounter
along the way, as you will see them early on, not only when working on
the last layer.

The rest of your application looks very good already, so I don't need to
comment on it :-)

-antrik-