[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?
From: |
Brandon Whaley |
Subject: |
Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2? |
Date: |
Tue, 2 Apr 2019 15:11:40 -0400 |
I think I see what the biggest point of confusion here is. When fab
executes a task but no -H parameters are set and the decorator
specifies no host, the Connection object is instead an invoke Context
object, which has no .local. I agree that that's not intuitive, but
I'm not sure where that choice is made in the fabric codebase.
On Tue, Apr 2, 2019 at 2:50 PM Michel Albert <address@hidden> wrote:
>
> Creating a new Context from the connection config did indeed work. So I am
> planning - for now - to go ahead with this.
>
> I hear you on the point of separating invoke from fabric tasks.
>
> To elaborate, and please don't read the following as a kind of rant, just as
> an elaboration where the need to mix remote & local tasks comes from.
>
> My current aim is to migrate an old, rather large and complex collection of
> fabric-1.x tasks to fabric-2.x And many of those tasks mix both remote and
> local commands. My aim is to keep everything available under the "umbrella"
> "fab" executable. Many tasks are shared among an ever growing team via a
> library made available to them. I really want to avoid having two executables
> where it may not be clear from the outside which task needs to be executed
> using invoke and which one needs fabric. The end-goal is to have a task
> runner which has the same core tasks for all of our projects. Namely:
>
> develop - to create a new development environment
> build - To build and bundle whatever is necessary for deployment
> test - To run unit-tests
> deploy - To deploy to a targeted environment
>
> We chose fabric in the past because it is really flexible, and lets us use it
> even for non-Python projects. And most of all it is extensible. So most
> projects have more than these 4 core tasks. Not being able to determine
> whether to use "inv" or "fab" would quickly become quite annoying.
>
> The library has grown over the past and it helped us a lot in our day-to-day
> business. Splitting the tasks now into an invoke- and fabric-library is
> definitely something I will take a closer look at, but I am afraid that some
> of the more complex tasks might lose in readability when doing so. I will see.
>
> In any case, your feedback helped me a lot. I wonder if this would be a
> useful example to put into the docs (that is,calling an invoke task from a
> fabric task). It does not seem to be something very common, but I had a lot
> of trouble finding a solution without falling back to this mailing list.
>
> On Tue, Apr 2, 2019 at 5:35 PM Brandon Whaley <address@hidden> wrote:
>>
>> Have you tried instantiating a context with the config from Fabric's
>> connection? I haven't mixed invoke tasks and fabric tasks in the same
>> module before, I'm surprised the same task decorator will work for
>> both using `inv` on the command line and `fab` as well. If you have a
>> task that only does local work inside your fabfile module, I'd
>> recommend using conn.local instead of making it an invoke task. If
>> you have an existing invoke task codebase, I would import that module
>> from my fabfile module and run tasks with a locally created context,
>> since you're not calling the task from the invoke cli.
>>
>> ```
>> tasks.py
>> ====
>> from invoke.tasks import task
>>
>> @task
>> def build(ctx):
>> ctx.run('make artifact')
>>
>> fabfile.py
>> ====
>> from fabric.tasks import task
>> from invoke.context import Context
>> from tasks import build
>>
>> @task(hosts=PROD)
>> def deploy(conn):
>> ctx = Context(config=conn.config)
>> build(ctx)
>> conn.put('artifact', '/app')
>> ```
>>
>> On Tue, Apr 2, 2019 at 2:04 AM Michel Albert <address@hidden> wrote:
>> >
>> > I understand the differences. The only traceback I got was the one I sent
>> > earlier. This traceback is correct, only maybe a bit cryptic.
>> >
>> > So far, I managed to get some basic tasks ported to fabric2.
>> >
>> > However, this morning I ran into a new problem:
>> >
>> > I have a remote task (deploy) which depends on a local task (build) and I
>> > want to be able to call the build step separately. But now I don't know
>> > how to call "build" from the "deploy" task. The docs state to simply call
>> > the function and forward the connection/context. But when calling from the
>> > "deploy" task, I have a "Connection" instance, but I need a "Context"
>> > instance to pass into the "build" task. Is there a way to get the Invoke
>> > context from the Connection instance? Example:
>> >
>> > @task
>> > def build(ctx):
>> > ctx.run('make artifact')
>> >
>> > @task(hosts=PROD)
>> > def deploy(conn):
>> > build(?) # <- what do I need to pass onto build? "conn" won't work
>> > conn.put('artifact', '/app')
>> >
>> > On Mon, Apr 1, 2019 at 9:33 PM Brandon Whaley <address@hidden> wrote:
>> >>
>> >> There is no local attribute on invoke contexts, only on fabric
>> >> connections (which inherits from invoke contexts). If this isn't what
>> >> you're trying to do, could you provide some code examples that give
>> >> you the traceback you're seeing?
>> >>
>> >> On Mon, Apr 1, 2019 at 1:07 PM Michel Albert <address@hidden> wrote:
>> >> >
>> >> > Ok. I think I will get it to work like that.
>> >> >
>> >> > But after fiddling around with it I found out that the type of the
>> >> > first argument depends on whether a "hosts" argument was passed into
>> >> > the task decorator or not. This was quite confusing. And the error is a
>> >> > bit cryptic. When calling "local" on an invoke context, the following
>> >> > AttributeError is thrown:
>> >> >
>> >> > Traceback (most recent call last):
>> >> > File "/usr/lib/python3.7/site-packages/invoke/config.py", line 113,
>> >> > in __getattr__
>> >> > return self._get(key)
>> >> > File "/usr/lib/python3.7/site-packages/invoke/config.py", line 178,
>> >> > in _get
>> >> > value = self._config[key]
>> >> > File "/usr/lib/python3.7/site-packages/invoke/config.py", line 169,
>> >> > in __getitem__
>> >> > return self._get(key)
>> >> > File "/usr/lib/python3.7/site-packages/invoke/config.py", line 178,
>> >> > in _get
>> >> > value = self._config[key]
>> >> > KeyError: 'local'
>> >> >
>> >> > During handling of the above exception, another exception occurred:
>> >> >
>> >> > Traceback (most recent call last):
>> >> > File "/home/exhuma/.local/bin/fab", line 10, in <module>
>> >> > sys.exit(program.run())
>> >> > File "/usr/lib/python3.7/site-packages/invoke/program.py", line 363,
>> >> > in run
>> >> > self.execute()
>> >> > File "/usr/lib/python3.7/site-packages/invoke/program.py", line 532,
>> >> > in execute
>> >> > executor.execute(*self.tasks)
>> >> > File "/usr/lib/python3.7/site-packages/invoke/executor.py", line 129,
>> >> > in execute
>> >> > result = call.task(*args, **call.kwargs)
>> >> > File "/usr/lib/python3.7/site-packages/invoke/tasks.py", line 128, in
>> >> > __call__
>> >> > result = self.body(*args, **kwargs)
>> >> > File "/home/exhuma/tmp/fabfile.py", line 8, in get_version
>> >> > version = ctx.local('python setup.py --version').strip()
>> >> > File "/usr/lib/python3.7/site-packages/invoke/config.py", line 125,
>> >> > in __getattr__
>> >> > raise AttributeError(err)
>> >> > AttributeError: No attribute or config key found for 'local'
>> >> >
>> >> > Valid keys: ['connect_kwargs', 'forward_agent', 'gateway',
>> >> > 'inline_ssh_env', 'load_ssh_configs', 'port', 'run', 'runners',
>> >> > 'ssh_config_path', 'sudo', 'tasks', 'timeouts', 'user']
>> >> >
>> >> > Valid real attributes: ['cd', 'clear', 'config', 'cwd', 'from_data',
>> >> > 'pop', 'popitem', 'prefix', 'run', 'setdefault', 'sudo', 'update']
>> >> >
>> >> >
>> >> > On Mon, Apr 1, 2019 at 5:37 PM Brandon Whaley <address@hidden> wrote:
>> >> >>
>> >> >> Hi Mich,
>> >> >>
>> >> >> The connection object (you're using ctx in your examples) has a .local
>> >> >> method that is just a pass-through to invoke.run. It's documented on
>> >> >> the connection object's page:
>> >> >> http://docs.fabfile.org/en/2.4/api/connection.html?highlight=local#fabric.connection.Connection.local
>> >> >>
>> >> >> On Mon, Apr 1, 2019 at 5:55 AM Michel Albert <address@hidden> wrote:
>> >> >> >
>> >> >> > Hi,
>> >> >> >
>> >> >> >
>> >> >> > Consider the following fabric-1 task. For illustration I kept it
>> >> >> > really short:
>> >> >> >
>> >> >> > @fab.task
>> >> >> > def sample():
>> >> >> > version = fab.local('python setup.py --version')
>> >> >> > fab.run('mkdir -p /snapshots/%s' % version.strip())
>> >> >> >
>> >> >> > This task needs to run a local and remote command. I am now trying
>> >> >> > to port this to fabric-2, and I can't figure out how I can implement
>> >> >> > this. If I define the "hosts" variable in the task, then the first
>> >> >> > line will be executed on the remote host as well, which I don't
>> >> >> > want. A naive aproach which won't work:
>> >> >> >
>> >> >> > @task(hosts=PROD)
>> >> >> > def sample(ctx):
>> >> >> > version = ctx.run('python setup.py --version').strip() # <-
>> >> >> > this won't work
>> >> >> > ctx.run('mkdir -p /snapshots/%s' % version)
>> >> >> >
>> >> >> > At first I thought I would split the task into two, one for just
>> >> >> > local commands and one for remote tasks, but then I am forced to
>> >> >> > pass in the context, which will in turn cause it again to be run
>> >> >> > remotely:
>> >> >> >
>> >> >> > @task
>> >> >> > def get_version(ctx):
>> >> >> > version = ctx.run('python setup.py --version').strip()
>> >> >> > return version
>> >> >> >
>> >> >> > @task(hosts=PROD)
>> >> >> > def sample(ctx):
>> >> >> > version = get_version(ctx) # <- this won't work
>> >> >> > ctx.run('mkdir -p /snapshots/%s' % version)
>> >> >> >
>> >> >> > How can I accomplish something like this? And where is it noted in
>> >> >> > the docs? In the current example on the "Upgrading from 1.x" page
>> >> >> > does not have a single task mixing local with remote commands in any
>> >> >> > way.
>> >> >> >
>> >> >> >
>> >> >> > Regards,
>> >> >> >
>> >> >> >
>> >> >> > Mich
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Fab-user mailing list
>> >> >> > address@hidden
>> >> >> > https://lists.nongnu.org/mailman/listinfo/fab-user
- [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Michel Albert, 2019/04/01
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Brandon Whaley, 2019/04/01
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Michel Albert, 2019/04/01
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Brandon Whaley, 2019/04/01
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Michel Albert, 2019/04/02
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Brandon Whaley, 2019/04/02
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Michel Albert, 2019/04/02
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?,
Brandon Whaley <=
- Re: [Fab-user] How do I combine "local" with "remote" tasks in fabric 2?, Michel Albert, 2019/04/02