Re: [Fab-user] Parallel operations for container creation

[Brandon Whaley]

Hi Ian,

Is there any reason you can't whip up a fab function to use execute() and the parallel decorator to perform this task?  If you want it to be re-usable without editing code, have it load a config file (json or otherwise) that describes the options to do per container.  Another strategy would be to use 'container-1,container-2,container-3' as your host list, then within your function use a 'with settings(host_string="host"):' block to change the host string dynamically before executing any run() functions.

Here's an example that I just whipped up to test on my own openvz node.  It will create containers named after the hosts you specify:

def create_container():

    container_name = env.host_string

    base_name, ctid = container_name.split('-')

    with settings(host_string="h4xvz"):

        run("vzctl create %s --hostname %s --ostemplate ubuntu-12.10-x86_64"%(

            ctid,

            container_name

        ))

        address = "10.105.0.%s" % ctid

        run("vzctl set %s --ipadd %s --nameserver 8.8.8.8 --save" % (

            ctid,

            address

        ))

        run("""echo ' >> /etc/vz/conf/%s.conf""" % ctid)

        run("""vzctl start %s""" % ctid)

Here's an example showing it in action:

$ fab -H container-101,container-102 -P create_container

[container-101] Executing task 'create_container'

[container-102] Executing task 'create_container'

[h4xvz] run: vzctl create 102 --hostname container-102 --ostemplate ubuntu-12.10-x86_64

[h4xvz] run: vzctl create 101 --hostname container-101 --ostemplate ubuntu-12.10-x86_64

[h4xvz] out: Creating container private area (ubuntu-12.10-x86_64)

[h4xvz] out: Creating container private area (ubuntu-12.10-x86_64)

[h4xvz] out: Performing postcreate actions

[h4xvz] out: Performing postcreate actions

[h4xvz] out: CT configuration saved to /etc/vz/conf/101.conf

[h4xvz] out: Container private area was created

[h4xvz] out: 

[h4xvz] out: CT configuration saved to /etc/vz/conf/102.conf

[h4xvz] out: Container private area was created

[h4xvz] out: 

[h4xvz] run: vzctl set 101 --ipadd 10.105.0.101 --nameserver 8.8.8.8 --save

[h4xvz] run: vzctl set 102 --ipadd 10.105.0.102 --nameserver 8.8.8.8 --save

[h4xvz] out: CT configuration saved to /etc/vz/conf/101.conf

[h4xvz] out: 

[h4xvz] run: echo ' >> /etc/vz/conf/101.conf

[h4xvz] out: CT configuration saved to /etc/vz/conf/102.conf

[h4xvz] out: 

[h4xvz] run: echo ' >> /etc/vz/conf/102.conf

[h4xvz] run: vzctl start 102

[h4xvz] run: vzctl start 101

[h4xvz] out: Starting container ...

[h4xvz] out: Container is mounted

[h4xvz] out: Starting container ...

[h4xvz] out: Container is mounted

[h4xvz] out: Adding IP address(es): 10.105.0.102

[h4xvz] out: Adding IP address(es): 10.105.0.101

[h4xvz] out: Setting CPU units: 1000

[h4xvz] out: Setting CPU units: 1000

[h4xvz] out: Container start in progress...

[h4xvz] out: 

[h4xvz] out: Container start in progress...

[h4xvz] out: 

Done.

And finally, here's my two new containers running on the host:

       101         21 running   10.105.0.101    container-101

       102         18 running   10.105.0.102    container-102

Naturally you would substitute the name of your host where I have h4xvz and modify this as needed for your environment.  I'm just giving a functional example of parallelism on a single endpoint by "cheating" how the host list works.

On Tue, May 14, 2013 at 8:14 PM, Ian Stokes-Rees <address@hidden> wrote:

I have read http://docs.fabfile.org/en/1.6/usage/parallel.html and many threads on this topic, but I thought I'd try in my own way to phrase what I believe must be a common situation for many people.

Short version:

For simplicity, imagine a case with 3 VZ containers that I am trying to provision on the same container host.  Right now I have to start 3 terminals, and I execute in each one of them:

fab -H host --name container-1 --index 1 --mode prod --user jdoe
fab -H host --name container-2 --index 2 --mode prod --user jdoe
fab -H host --name container-3 --index 3 --mode prod --user jdoe

I would much rather execute something like:

fab -H host -P --name container-1, container-2,container-3 --index 1,2,3 --mode prod --user jdoe

And have fab auto-magically map that one-liner to the 3 commands above and run them in parallel.  That *seems* like a reasonable request, but I just can't find a way to make that happen.  It is important enough for my work that I'd be willing to write some @map decorator or similar to facilitate an implementation.

For a few more details, each container creation takes ~15 minutes.  They parallelize almost perfectly, so rather than waiting 5 hours for 20 containers to all come up serially (20 x 15 minutes), I'd rather wait 15 minutes.  There are some slight parameter differences for each container.  Right now I need to start 20 terminal sessions and execute 20 slightly different commands.  Error prone!

TIA

Ian

_______________________________________________
Fab-user mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/fab-user