RE: cfexecd -D / rule pruning

[Martin, Jason H]

>I need more info to make sure you couldn't accomplish the same thing 
>without multiple calls for cfexecd.

The situation is thus: I have a set of machines that act much like
cluster servers. As such they have many directories mounted under some
common directory that will move between physical hosts on a day-to-day
basis, and more may be created at any time. There are quite a few of
these mounts per host.  Scale here is important, so lets say that
between just a couple of servers I have of 600 of these directories that
host Apache instances, 100 that host Oracle instances and 200 that host
other applications (as an example).

My goal is to be able to treat each of those directories as a host and
apply rules to them. So, (made up example) if the directory has a
/$(cluster_dir)/etc/httpd.conf in it, I'd like
/var/cfengine/repository/robots.txt copied to
/$(cluster_dir)/html/robots.txt. I'd also like to ensure that
/$(cluster_dir)/bin/httpd is running and that
/$(cluster_dir)/logs/access.log is rotated properly.

If each of these directories were a different physical host, I could use
modules, classes, and groups to accomplish this with a minimum of
typing. I could say that 'host' (meaning a directory under the basedir,
e.x. /mnt) A,B,C,and D should get one version of robots.txt, and
F,G,H,I, and J should get another, and all directories that have a file
named /$(cluster_dir)/logs/access.log should have it rotated. However,
when all of this is on one physical machine there isn't any way to
express this short of explicitly writing out a stanza for every
directory. Since I don't know until runtime the number and names (other
then the parent directory) it is impossible.  I really just want to be
able to apply a set of rules iteratively to a list of directories.

The _ideal_ way in my mind to do this would be to have a module create a
list of these directories then have a method iterate over them.
Unfortunately, method iteration does not exist. 

My workaround is to create a separate set of config files that will
apply to these directories as a whole. I can then have my top-level
Cfengine do physical-host specific activities and then run a
shellcommand that calls the 2nd level CFEngines via a shell script. The
script exports the environmental variable CFINPUTS to point to the
separate config directory and gets a list of directories under the
top-level mountpoint. For each of those directories it stores the
directory name in a temporary file, then calls cfagent -D `basename
$THEDIR`.  A module within the 2nd-level Cfengine will read the
temporary file and define the variable $(cluster_dir). All of the rules
in the 2nd-level configuration are written to be relative to
$(cluster_dir). This way I can make changes to 200 Apache instances
spread across multiple machines with just 1 copy statement, regardless
of which machine is hosting that directory on a given day. I can also
utilize groups like I would hosts to classify groups of directories,
instead of individual copy statements. 

So, what I want to do is possible if CFENGINE #1 calls cfagent. However,
I also have tools that process the emails resulting from cfexecd output,
and a nested CFEngine's output will be much harder to parse. So, I'd
rather that CFENGINE #2 generate its own email to my tools. The problem
here is that 'import' optimizes away all the rules that relate to the
directories since the class is not yet defined. I can't know ahead of
time the set of directories that will exist and so cannot add them to
AddInstallable.  One solution I am trying is to modify the IsInstallable
function in eval.c to always return true and have a module define the
'host', however that does not feel like the best way. It would be much
simpler if I could call cfexecd to define the current 'host' (aka
`basename $directory`) with the -D option to cfagent or iterate over a
method.

Thank you,
-Jason Martin 

-----Original Message-----
From: Christian Pearce [mailto:pearcec@perfectorder.com] 
Sent: Wednesday, August 10, 2005 1:27 PM
To: Martin, Jason H
Cc: help-cfengine@gnu.org
Subject: Re: cfexecd -D / rule pruning


Do you have time to jump into #cfengine?

if not I would need more info to make sure you couldn't accomplish the
same thing without multiple calls for cfexecd.  I don't know if there is
a "reason" other than oversight.  I wanted this in for quite sometime. I
worked on the code at a certain point.  I don't think it involved a
whole lot.

Mark might have a reason for not doing it.  There might even be a bug
open about this.

On Wed, 2005-08-10 at 12:59 -0700, Martin, Jason H wrote:
> Is there any particular reason that cfexecd doesn't allow arbitrary
> flags to be passed to cfagent?  cfrun lets you do a "-- -D someclass" 
> to pass arguments to cfagent, but this doesn't work for cfexecd. I 
> need to do this as I want to run a set of rules once per directory on 
> a given host, and there is no way to iterate most cfe rules over a 
> given set of destinations.  Instead, I am iterating CFE entirely over 
> that set of directories.
>  
> I need to define a class with the name of the directory. I would do it
> via a module except that the directory name is not known at 
> rule-writing time, so CFE is pruning away all of the rules for the 
> class before the module runs.
>  
> Another way to approach the problem is to ask if it is possible to
> disable the pruning of rules at parse time.  The effect would be 
> similar to adding every possible class to AddInstallable.
>  
> Thank you,
> -Jason Martin
> _______________________________________________
> Help-cfengine mailing list
> Help-cfengine@gnu.org
> http://lists.gnu.org/mailman/listinfo/help-cfengine