[Fab-user] Launch many parallel tasks against a single (local) host

[Tyler Pirtle]

I realize this use case in particular may sound a little strange, but bear with me.

I've got 10 files lets say, locally, and would like to launch 10 tasks to stage them to some other target. I'd like to do this in parallel, since the files are named appropriately (they're sharded), and what I'd like to do is to construct a call in fabric that executed some function over the same host with varying input.

At a first pass, I thought I would just try to get one task to execute N times in parallel at localhost:

@parallel

def Stuff():

  print "Sleep!"

  time.sleep(2)

$ fab -H localhost,localhost,localhost Stuff

[localhost] Executing task 'Stuff'

Sleep!

Done.

This doesn't work because the host list gets merged (merge in task_utils.py).

But, merge() isn't that smart.

$ fab -P -H address@hidden,address@hidden,address@hidden Stuff

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

Sleep!

Sleep!

Sleep!

So this is a workaround and a pretty silly hack - does anyone have a better way to do this? As a further hack, I wrap it in an easy-to-use command:

@parallel

def Stuff():

  print "Sleep - %s" % env.host

@hosts("localhost")

@parallel

def PLaunch(target="", times=1):

  h = []

  for i in xrange(int(times)):

    h.append("address@hidden" % i)

  execute(target, hosts=h)

$ time fab -f fab2.py -P PLaunch:Stuff,10

[localhost] Executing task 'PLaunch'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

address@hidden Executing task 'Stuff'

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Sleep - address@hidden

Done.

real 0m5.522s

user 0m0.352s

sys 0m0.251s

So i can strip off everything up to the @ and use that as the arg. Any better ideas? ;)

Thanks,

Tyler