Grand Unified System Installer (hurd distro) + my taughts and comments - new to hurd contributions, only taking my head out of the sand now.

[Ivan Malone]

Greetings All.


This will be a long post. And I hope I dont bore you. This is my first
attempt at contributing in a meanfull and full-time way to Hurd and GNU
in general. Have patience with me :D


I understand that the Hurd uses the common dpkg (apt-get install
whatever) for package management (although i suppose i could use any
package manager if i worked at it) and I just had a few questions to
which I couldn't find satisfactory answers to on google, debian or
ubuntu forums. I didn't want to post to their mailing lists because my
questions are more hurd related and I'm still in the "thinking" and
learning phase of my plans on contributing to the Hurd. 


Some idea's I wanted to implement a few years back before I knew more of
Hurd than its existence just wasn't possible in Linux but since comming
to Hurd I've found that if I had been using Hurd back when I had the
origonal taughts I could have made a start on it then, instead of now. 


Anyone, I wanted to share my taughts in the hope of solicting some
responces from some of you. 

Let me just first say that I have been shadowing the mailing lists now
for a while, I've read hundreds of pages on the GNU site, blogs, forums,
and I read every post to the bug-hurd and some debian, ubuntu,
kernel-hackers and Gentoo and if anything I ask seems stupid or answered
somewhere else or my train of taught is completly off the mark and I've
somehow passed over something relevent then forgive me. I'm not there
yet but I'm getting there. Been a very lazy programmer and stuck in my
ways the change from commericial programming to pure open-source in
everything I do is teaching this old dog new tricks. 

I've managed to get Hurd built from source many times (and killed it as
often) and I'm really enjoying migrating from linux to the ideas
presented in Hurd. Kudos to your work. I find it hard to believe you
dont have more developers because Hurd seems a natural progression (in
my opinion) from the current style kernels and I find it hard to
understand the resistance I get from my friends and co-workers whenever
I bring up the Hurd. (especially when I mention hurd was around before
linux.. jeez. even i knew that!) 

So, if you could endulge me a bit I'd be gratefull. This will be my
first and hopefully not last post here. 

1) Is the reason for using dpkg on hurd atm to manage packages because
of the hurds current association with Debian, in that they use it in
Debian and seen as they are supporting hurd it was a natural
progression? Or just maybe because something was needed and apt was
there waiting to be used. Or do you find yourself installing your own
packages by hand and compiling what you need yourselves the majority of
the time.

2) Is the GNU Package Management system thats under development realy on
the right track in your opinion? Do you see it been the final package
management system on hurd... given some of the ultimate goals and idea's
that Hurd is attempting to achieve (from what i understand). I wont go
into specifics but just interested in opinions in the most general
sense. 

3) I ask this because I'm becomming enthralled with filesystems, storage
of code (git and subversioned), PGP crytography, also unionmount
imparticular, additionally I love how everything in hurd is a service of
one kind or another and exporting interfaces for communication and
manipulation. The seperation between user (userspace) and kernel space
is more clearly defined (at least in my head) and infintly more
configurable in subtle ways that isn't immediatly obvious to someone
like myself comming from linux. Such as the level to which you can
change the entire user environment seperatly from system level to how
you can communicate with other programs and libraries or services. 

3) The Hurd is UNIX (POSIX) compatible - have you found anything in the
current draft POSIX.2 (i believe) standards that don't seem to provide
enough room, or perhaps, restrict development on Hurd in anyway, like
implenentation of future ideas? Or is Hurd only concerned with the
current POSIX standard in that it suits hurd at the moment to stick to
it. 

One aspect of hurd that I want to get involved in is File Systems. I
love that everything is decentralized. It makes no difference to me if
the objects are remote or localy hosted. To this end I've been following
unionmount related bugs/posts on this mailing list and looking at the
source with to be honest, is taking me longer than expected to get to
grips to due to assumptions and misunderstandings on my part. 

4) What kind of filesystem to you envision Hurd finally using as its
default, I dont see it been ext2 or ext3. There great but something (i
dont know what yet) is niggeling away at me and I'm not satisifed with
it. Especially when it matters not a wit to the user what filesystem he
is using. 

Another beast I've changed over to using almost exclusivly is GIT. Its
beautiful in its uglyness :D.

Apt, Portage (gentoo package management) and the GNU Package Manager are
all topics I'm looking into. Mikel Olasagisti (excuse if i spelt it
wrong) was working on Gentoo-Hurd and I too have an interest in using
portage (or something new based on portage/apt mix) but I believe that
development has stalled or stopped but I'll be going back over it soon
and hopefully get his and others taughts. Gentoo is not my distro of
choice but I like its philosphy of installing from source. 

Here is my train of taught and whats bugging me

Given the way Hurd uses translators and objects and exports interfaces
and everything is a service and it doesn't really matter a wit if its
remote or local I was thinking there has to be a better way to manage
all of this. Forgive me for not making my taughts more clear but I'm
thinking High Level at the minute and not gotten down to technical
details yet till I've worked out exactly whats bugging me and what I
want to achieve. 

With regard to Package Management and GIT - I think it should be much
easier and transparent for a developer to commit work to and from GIT,
idealy not even knowning or caring if its GIT, CVS or whatever
subversion system is in use and been able to use it much more naturally
with whatever development environment they choose - also - from a
package retrieval point of view it should be much more natural for a
user to be able to retrieve latest builds, stable versions, backports,
documentation, source code and so on. 


I've also noted that maybe we should be using the GNU Package system
where possible (i'm not sure to what degree its development is along yet
as I started investigating it 2 days ago)

I like apt on my debian and ubuntu systems. But I also like portage and
would like a mix between the two but used in sush a way as it could
possibly manage GIT, CVS etc through the likes of translators similiar
to how unionmount manages different filesystems under a commonly
understood theme. 

I was also wondering in general about the future of operating systems in
general. For example, ten years from now I'm predicting that the
dividing line of which operating system you use will be irrelevent when
it comes down to making a choice between applications for databases,
games, desktop publishing, multimedia, whatever. Mono, Java, Google, IBM
are all on to big things but again, something just doesn't seem right to
me about it all. I think the role of the operating system will come down
to just managing the local computer, providing means to communicate with
devices (weither remote or local) and the actual software the majority
of people use will either be service based (ie: running remotly
somewhere with some interface to connect to) or universal like Java,
Mono (notice i didn't say .net, microsoft I believe will become a big
player in open source (if reluctently). With that in mind I want to keep
reminding myself whatever I'm trying to figure out will have to be
easily extended. 

I was thinking about how Hurd treats everything as objects, interfaces,
services, translators - what union mount was designed to do, how package
managers work and the goals each wants to achieve, advantages,
disadvantes of each and suitability for given tasks. Also, the day to
day tasks of users and developers – whats common amoung them – whats the
differences – how can easy do their thing but exploit the other kind of
user to further development and encourage better bug reporting or
contributions. 


I was thinking of the amount of code that will have to change or become
redundent or stale or might not even have a place in an Object/Service
environment, at least from a higher point of view. OpenSSH, PGP and
Authentication should play a larger more transparent part in everyday
data transmission and user authentication. Unsecured communication needs
to be a thing of the past. Telnet has its uses but SSH is by far better.
Ping is essential networking tool – but for example, in a hurd system
with clustered machines, remote cpu interfaces, remote network devices
etc ping should be able to ping <this machine> from <these machines> and
<report the results to this interface (email, telnet, ssh, tty session,
file, whatever) for example as easily as you would ping <ip address>.
Hurds ideas have far reaching effects for even the most taken for
granted tool should it ever reach stable production level. 

Emails, Mailing list, Instant Messengers, VoIP, Web Browsing - they all
form a means of communication - but what if someone never uses Mailing
lists or forums, what if your friend isn't using an instant messenger,
what if this or that - surly there is a better way to some how merge all
these mediums so that that its irrilevent what the medium is.
Communications will continue to evolve, whats the best way to have it
all and leave room to keep expending. Sendmail is included with every
unix system, but should it not be possible somehow to exploit other
services on other systems without having the software installed on your
system or even caring if you do or not. It should not cost you a taught,
you shouldn't have to wonder if you need this or that to do whatever, it
should just be there. 


For example. In terminal. 

~/lynx http://www.gnu.com


The program 'lynx' is currently not installed. You can install it by
typing:

sudo apt-get install lynx


This is grand. And you can type apt-get install lynx and off you go. I'm
not saying it should be installed by default but I am saying that where
a known command was not found it should instead do something like this


~/lynx http://www.gnu.com


The 'lynx' interface is not up or the lynx software is not installed on
your system. Would you like to bring it up/install it now?
[u=up/i=install/c=cancel] - u (choose to bring it up)


When the service is brought up the interface locally would using hurd
mechinics connect to some remote location to provide the lynx program. 


Ok, lynx might have been a bad example but the point I'm trying to make
is I envision hurd been far simplier than current simple (once you've
learned them) methods. 


My first task that I've already been working on is a way to achieve a
workable booting Hurd system (not debian style, hurdports) for everyone.
Creating a Live CD, or forking a new distribution to installing on a
single or hundreds of machines should be as easy as 1,2,3 - download,
prepare, install. 

So far I've a few hundred (ugly) looking scripts and some c code to
manage dependancies, compiling, downloading, checking, configuration etc
but obviously this will need to evolve beyond scripting to something
more real and dependable. 

I'm very keen on using whats already available - modify what doesn't
quite suite - merge different projects that may not have had relation
before - to achieve this, hopefully without having to fork anything - i
think this is achievable through translaters and interfaces and would
suit a hurd environment very well (but should be executable from any
POSIX system). 

Right now the installation depends on apt and git and runs on debian
linux but I'd like the whole installation system itself to be hurd
based.


What I hope to get from the first stage is a basic system that can be
booted from Live CD or USB or maybe booted over the network.


The Live CD will be universal. In a sense it is not a distro in itself
but a universal tool. It will ideally be run on Hurd. Hurd will allow
the user to choose what Linux/BSD Distribution he wants to install or
configure including the latest Hurd from source. 


If he chooses a distribution that does not exist cached somewhere
locally (on the cd, on the network, on local storage) then it will
download it from either A) Repository (install from source), B) Direct
download from HTTP or whatnot. It should not be biased toward any
distribution though offically in my mind, though it is Hurd I want it
for. 


What do I need this for when I can download and install from the usual
sources.... mostly because that is exactly what I dont do. Mostly, I
download the source, create a basic environment, compile everything in
along the lines of portage, get the package mananger up – setup the
repository locations and execute scripts to download, compile and
install what I need. It only goes to the likes of binaries when I'm
using stable releases or not interested in the source. I normally use my
own locally mirrored sources (currently this is in the range of 5tb at
work) and includes the full debian, ubuntu, linux mint, gentoo and
sabyon repositories. 


Obviously, this is beyond the average user or indeed most users who have
no need or use for this method – so for them it would be a minimal hurd
system booted from grub that will use Unionmount and virtualization to
mount a location and environment for the other distro and begin its
normal install procedure (or clone to/from disk, partition). Requires
about an extra 50 megs on disk (after the basic hurd install) and boot
time is slowed down by about 15 seconds on most machines during the
inital CD boot. 


Would it be possible in Hurd... from a users environment (installuser
account) to setup the envirnment in such a way as a system mounted and
booted into it (such as ubuntu) could not see the underlying Hurd and
continue as normal? - ie: advanced chroot in the broadest loosest sense/


Advantages... failed install wont bring down the system and I can
immediatly go into debugging or recovery mode without having to reboot,
reburn another cd for example. 


The Other Distro wouldn't have to worry about device drivers as the Hurd
system would provide compatibility layers through interfaces enough for
the other system to work using its stock drivers for the likes of disk,
network etc. - but of course, provide some method for the other distro
to loads its only specific drivers if need be. This could possibly
decrease the size of potentional distributions live cd's as it wouldn't
need so much to get up and running, the grub/Hurd combo would take care
of it (and provide a real world, working example of a hurd system to
begin with – a proof of concept that hurd is real, useable and not just
a hackers toy). 


Another problem is enouraging lesser skilled or non-skilled people to
contibute... but in a usefull way. We all know and accept by now I think
that you can't teach some people. Majority of bug reports are useless,
they make assumptions that are not true or use the wrong tool for the
wrong job or ask the wrong question in the wrong place about something
thats not relivent to their problem to begin with (or is this only my
experience).


For most the prospect of using GIT, Launchpad or making modifications to
this or that is daunting when they dont know how - there should be a
system in place with the likes of documention, bugs, suggestions,
requests, should be automated by manageable. 

This as a first stage. After a base distribution is installed with all
required services and systems in place then what? 


I think its time the Hurd took the spot light. I know its not stable. I
know its not production ready. But I think in a limited context it can
be. 


So, after all this (thanks for reading to the end by the way), I'd love
your taughts on the future of hurd, how you see it working in the real
world, in general non-too-technical-sense and terms.


What I'm creating at the minute is the following:

      * A base hurd install booted from either LiveCD, Usb or Network
        source. (of course i have to make some allowances for size and
        purpose)
        
      * A terminal and basic tools
        
      * A method to install (and update) a Hurd system (xwindows too)
        that everyone use. This wont be like Debian Hurd or Gentoo Hurd.
        I would prefer if this was just GNU/Hurd. Using only GNU tools
        or 100% compatible GPL's GNU software. But I dont have the
        skills, time or patience to manage such a thing and you are all
        working to that end anyway in one way or another. What I want is
        just a Hurd system thats not tied or relying upon another
        Project but would have good relations with other Hurds.
        
      * The LiveCD once booted should be able to install either Hurd or
        another distro – think of it like you would Grub, grub is the
        Grand Unified Boot Loader – My Hurd I guess would be the Grand
        Unified System Installer (gusi?!?!? uck. no. Need a better
        name).
        


I'm hoping that, with enough people using it for that purpose I can help
get it stable and production ready in the sense that it does this job
well, everytime and it must be able to inherit new changes easily. Once
that is done I hope that I can build upon it to such a degree that
everyone can get a standard very basic hurd system up and running, we
dont need GNOME or KDE or fancy effects for production systems.


Other things, beyond this goal and out of scope I'm looking into along
the lines of file systems is DragonFly BSD's Hammer file system. I'm
interested in porting this to Hurd and expect to begin real work on it
later this year once I get the LiveCD Working.


I have definat ambitions with regard to hurd and I've been taking it
very seriously over the course of this year since I began thinking of
Hurd as a possible platform.


So, to finish, and summerize - These are all in the works in one degree
or another.


LiveCD Hurd (grand unified system installer) – Gusi.

Base Hurd System (no name yet, was toying with Ostrix)


Hurd based Bootable recovery Partition for all our clients machines
(recovery, backups, cloning, emergency recovery)


Porting Hammer to Hurd. 


Package Manager Specification for Hurd (like portage, gnu, or apt)


Anyway, taughts, critisims, direction, advice, comments are more than
welcome.


Thank you for reading... I hope it wasn't too painfull (and if you used
pg/dn shame on you)


Regards,

Ivan