Re: [Fab-user] Launch many parallel tasks against a single (local) host

[Morgan Goose]

What is it you even mean by staging? You might be over looking GNU
commands that will do exactly what you need. If you give us some more
information, we might be able to assist.

-goose

On Fri, Mar 23, 2012 at 7:42 PM, Tyler Pirtle <address@hidden> wrote:
> I realize this use case in particular may sound a little strange, but bear
> with me.
>
> I've got 10 files lets say, locally, and would like to launch 10 tasks to
> stage them to some other target. I'd like to do this in parallel, since the
> files are named appropriately (they're sharded), and what I'd like to do is
> to construct a call in fabric that executed some function over the same host
> with varying input.
>
> At a first pass, I thought I would just try to get one task to execute N
> times in parallel at localhost:
>
> @parallel
> def Stuff():
>   print "Sleep!"
>   time.sleep(2)
>
>
> $ fab -H localhost,localhost,localhost Stuff
> [localhost] Executing task 'Stuff'
> Sleep!
>
> Done.
>
> This doesn't work because the host list gets merged (merge in
> task_utils.py).
>
> But, merge() isn't that smart.
>
> $ fab -P -H address@hidden,address@hidden,address@hidden Stuff
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> Sleep!
> Sleep!
> Sleep!
>
> So this is a workaround and a pretty silly hack - does anyone have a better
> way to do this? As a further hack, I wrap it in an easy-to-use command:
>
>
> @parallel
> def Stuff():
>   print "Sleep - %s" % env.host
>
> @hosts("localhost")
> @parallel
> def PLaunch(target="", times=1):
>   h = []
>   for i in xrange(int(times)):
>     h.append("address@hidden" % i)
>   execute(target, hosts=h)
>
> $ time fab -f fab2.py -P PLaunch:Stuff,10
> [localhost] Executing task 'PLaunch'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> address@hidden Executing task 'Stuff'
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
> Sleep - address@hidden
>
> Done.
>
> real 0m5.522s
> user 0m0.352s
> sys 0m0.251s
>
>
> So i can strip off everything up to the @ and use that as the arg. Any
> better ideas? ;)
>
> Thanks,
>
>
> Tyler
>
>
> _______________________________________________
> Fab-user mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/fab-user
>