Re: [Fab-user] Launch many parallel tasks against a single (local) host

[Morgan Goose]

Use xargs:
def parallel_local(cmd, list_of_args, procs=4):
    local("echo '%s' | xargs -P %d -n1 %s" % (" ".join(list_of_args),
num_of_procs, cmd))


I find that most everything you wanna do has a gnu command or flag for
it already. thankfully.

-goose


On Tue, Mar 27, 2012 at 2:57 PM, Tyler Pirtle <address@hidden> wrote:
>
>
> On Tue, Mar 27, 2012 at 1:13 PM, Morgan Goose <address@hidden>
> wrote:
>>
>> What is it you even mean by staging? You might be over looking GNU
>> commands that will do exactly what you need. If you give us some more
>> information, we might be able to assist.
>>
>> -goose
>>
>
> Hi goose - let me try and clarify.
>
> My use case is something like invoking a function over one host many times
> with
> different argument values (Or many hosts with many values for extra credit).
>
> So I've got some function i want to execute on localhost,
>
> def foo(i): ...
>
> What I'd like to do is imap(foo, xrange(1, 10)) in parallel.
>
> The execute() function doesn't give me a way to express this (the way I've
> done this below is to generate fake host lists). The host list seems to be
> the unit controlling the parallelism, I'd like to be able to control it via
> input arguments
> as well.
>
> To illustrate:
>
> (with env.parallel = True)
> hosts = ["localhost", "localhost", "localhost"]
> execute(foo, hosts, ... )
>
> Would execute "foo" 3 times in parallel (except it wouldnt because host
> lists are uniq'd)
>
> What I'm looking for:
>
> hosts = ["localhost"]
> input = xrange(10)
> execute(foo, hosts, input)
>
> Should be equivalent to imap(foo, input) over hosts.
>
>
>
>
>>
>> On Fri, Mar 23, 2012 at 7:42 PM, Tyler Pirtle <address@hidden> wrote:
>> > I realize this use case in particular may sound a little strange, but
>> > bear
>> > with me.
>> >
>> > I've got 10 files lets say, locally, and would like to launch 10 tasks
>> > to
>> > stage them to some other target. I'd like to do this in parallel, since
>> > the
>> > files are named appropriately (they're sharded), and what I'd like to do
>> > is
>> > to construct a call in fabric that executed some function over the same
>> > host
>> > with varying input.
>> >
>> > At a first pass, I thought I would just try to get one task to execute N
>> > times in parallel at localhost:
>> >
>> > @parallel
>> > def Stuff():
>> >   print "Sleep!"
>> >   time.sleep(2)
>> >
>> >
>> > $ fab -H localhost,localhost,localhost Stuff
>> > [localhost] Executing task 'Stuff'
>> > Sleep!
>> >
>> > Done.
>> >
>> > This doesn't work because the host list gets merged (merge in
>> > task_utils.py).
>> >
>> > But, merge() isn't that smart.
>> >
>> > $ fab -P -H address@hidden,address@hidden,address@hidden Stuff
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > Sleep!
>> > Sleep!
>> > Sleep!
>> >
>> > So this is a workaround and a pretty silly hack - does anyone have a
>> > better
>> > way to do this? As a further hack, I wrap it in an easy-to-use command:
>> >
>> >
>> > @parallel
>> > def Stuff():
>> >   print "Sleep - %s" % env.host
>> >
>> > @hosts("localhost")
>> > @parallel
>> > def PLaunch(target="", times=1):
>> >   h = []
>> >   for i in xrange(int(times)):
>> >     h.append("address@hidden" % i)
>> >   execute(target, hosts=h)
>> >
>> > $ time fab -f fab2.py -P PLaunch:Stuff,10
>> > [localhost] Executing task 'PLaunch'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > address@hidden Executing task 'Stuff'
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> > Sleep - address@hidden
>> >
>> > Done.
>> >
>> > real 0m5.522s
>> > user 0m0.352s
>> > sys 0m0.251s
>> >
>> >
>> > So i can strip off everything up to the @ and use that as the arg. Any
>> > better ideas? ;)
>> >
>> > Thanks,
>> >
>> >
>> > Tyler
>> >
>> >
>> > _______________________________________________
>> > Fab-user mailing list
>> > address@hidden
>> > https://lists.nongnu.org/mailman/listinfo/fab-user
>> >
>
>