[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Fab-user] rsync_project
From: |
Christian Vest Hansen |
Subject: |
Re: [Fab-user] rsync_project |
Date: |
Fri, 3 Apr 2009 23:12:46 +0200 |
On Fri, Apr 3, 2009 at 11:01 PM, Paul Baumgart <address@hidden> wrote:
> Yeah, I took the change you pushed and added to that support for more than
> one exclude option, because otherwise the exclude parameter isn't nearly as
> useful as it could be.
Ok, applied. Thanks!
>
> Sorry if I wasn't clear.
>
> Paul
>
> On Fri, Apr 3, 2009 at 1:41 PM, Christian Vest Hansen <address@hidden>
> wrote:
>>
>> I think you misunderstood me!
>>
>> I had already pushed a change. Or maybe you're building on top of that?
>>
>> On Fri, Apr 3, 2009 at 10:34 PM, Paul Baumgart <address@hidden> wrote:
>> > Ok, fair enough. :-)
>> >
>> > How's this then:
>> >
>> > @operation
>> > def rsync_project(remotedir, exclude=[], delete=False, extra_opts='',
>> > **kwargs):
>> > """
>> > Uploads the current project directory using rsync.
>> > By using rsync, only changes since the last upload are actually sent
>> > over
>> > the wire, rather than the whole directory like when using
>> > upload_project.
>> >
>> > Requires the rsync command-line utility to be available both on the
>> > local
>> > and the remote machine.
>> >
>> > Parameters are:
>> > remotedir: the directory on the remote machine to which
>> > to
>> > rsync the current project. The project
>> > directory
>> > becomes a subdirectory of the remotedir.
>> > exclude (optional): values passed to rsync's --exclude option.
>> > If the parameter object is iterable, (that
>> > is,
>> > it
>> > defines an __iter__ method), each of its
>> > elements is
>> > passed to a separate --exclude option.
>> > Otherwise,
>> > the object is passed as a string to a single
>> > --exclude option.
>> > See the rsync manpage for details on
>> > specifying
>> > filter rule arguments for --exclude.
>> > delete (optional): True or False, whether to delete remote
>> > files
>> > that
>> > don't exist locally. Defaults to False.
>> > extra_opts (optional): Additional command-line options to set
>> > for
>> > rsync.
>> >
>> > The rsync command is built from the options as follows:
>> > rsync [--delete] [--exclude exclude[0][, --exclude[1][, ...]]]
>> > \\
>> > -pthrvz [extra_opts] ../<project dir>
>> > <fab_user>@<host>:<remotedir>
>> > """
>> > username = ENV.get('fab_user')
>> >
>> > if not hasattr(exclude, '__iter__'):
>> > exclude = [exclude]
>> >
>> > exclude_opts = ' --exclude "%s"' * len(exclude)
>> > exclusions = tuple([str(s).replace('"', '\\\\"') for s in exclude])
>> >
>> > options_map = {
>> > "delete" : '--delete' if delete else '',
>> > "exclude" : exclude_opts % exclusions,
>> > "extra" : extra_opts
>> > }
>> > options = "%(delete)s%(exclude)s -pthrvz %(extra)s" % options_map
>> > cwd = '../' + os.getcwd().split(os.sep)[-1]
>> > userhost = "$(fab_user)@$(fab_host)"
>> > rdir = _lazy_format(remotedir, ENV)
>> >
>> > cmd = "rsync %s %s %s:%s" % (options, cwd, userhost, rdir)
>> > local_per_host(cmd, **kwargs)
>> >
>> >
>> > Besides reducing the length of lines, I also made it more pythonic by
>> > using
>> > duck typing to determine how to handle the exclude parameter. So now you
>> > can
>> > pass integers, complex numbers, or anything else with a __str__ method
>> > and
>> > it will work.
>> >
>> > I also cleaned up the documentation a bit and made some parts of it more
>> > precise.
>> >
>> > Paul
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Apr 3, 2009 at 10:50 AM, Christian Vest Hansen
>> > <address@hidden> wrote:
>> >>
>> >> Good idea. But as you may have guessed, I'm not fond of long lines :)
>> >>
>> >> On Fri, Apr 3, 2009 at 5:21 AM, Paul Baumgart <address@hidden> wrote:
>> >> > Hi Christian,
>> >> >
>> >> > Thanks, your version looks a lot cleaner.
>> >> >
>> >> > I would like to add one thing, because I think it makes the exclude
>> >> > parameter much more useful:
>> >> >
>> >> > @operation
>> >> > def rsync_project(remotepath, exclude=False, delete=False,
>> >> > extra_opts='',
>> >> > **kwargs):
>> >> > """
>> >> > Uploads the current project directory using rsync.
>> >> > By using rsync, only changes since last upload are actually sent
>> >> > over
>> >> > the wire, rather than the whole directory like using
>> >> > upload_project.
>> >> >
>> >> > Requires the rsync command-line utility to be available both on
>> >> > the
>> >> > local
>> >> > and the remote machine.
>> >> >
>> >> > Parameters are:
>> >> > remotepath: the path on the remote machine to which to
>> >> > rsync
>> >> > the
>> >> > current project
>> >> > exclude (optional): values passed to rsync's --exclude option.
>> >> > If the parameter is iterable, each of its
>> >> > elements
>> >> > is passed to a separate --exclude
>> >> > argument.
>> >> > If
>> >> > the
>> >> > parameter is a string, that string is the
>> >> > value
>> >> > passed to the single --exclude argument.
>> >> > See the rsync manpage for details on using
>> >> > the
>> >> > --exclude option.
>> >> > delete (optional): True or False, whether to delete remote
>> >> > files
>> >> > that
>> >> > don't exist locally.
>> >> > extra_opts (optional): Additional command-line options to set
>> >> > for
>> >> > rsync.
>> >> >
>> >> > The rsync command is built from the options as follows:
>> >> > rsync [--delete] [--exclude exclude] -pthrvz [extra_opts] \\
>> >> > <project dir> <fab_user>@<host>:<remotepath>
>> >> > """
>> >> > username = ENV.get('fab_user')
>> >> >
>> >> > if isinstance(exclude, basestring):
>> >> > exclude = [exclude]
>> >> >
>> >> > options_map = {
>> >> > "delete" : '--delete' if delete else '',
>> >> > "exclude" : exclude and ' '.join(['--exclude "%s"' %
>> >> > opt.replace('"',
>> >> > '\\\\"') for opt in exclude]) or '',
>> >> > "extra" : extra_opts
>> >> > }
>> >> > options = "%(delete)s %(exclude)s -pthrvz %(extra)s" % options_map
>> >> > cwd = '../' + os.getcwd().split(os.sep)[-1]
>> >> > userhost = "$(fab_user)@$(fab_host)"
>> >> > rpath = _lazy_format(remotepath, ENV)
>> >> >
>> >> > cmd = "rsync %s %s %s:%s" % (options, cwd, userhost, rpath)
>> >> > local_per_host(cmd, **kwargs)
>> >> >
>> >> > So, instead of taking just a string, it can take either a string or a
>> >> > list,
>> >> > and multiple --exclude options are generated from a list. A string
>> >> > parameter
>> >> > is treated just as it is currently.
>> >> >
>> >> > Paul
>> >> >
>> >> > On Thu, Apr 2, 2009 at 12:38 PM, Christian Vest Hansen
>> >> > <address@hidden> wrote:
>> >> >> Ok. I implemented this change, but I modified it a little bit.
>> >> >>
>> >> >> On Thu, Apr 2, 2009 at 9:17 PM, Christian Vest Hansen
>> >> >> <address@hidden> wrote:
>> >> >>> On Sun, Mar 22, 2009 at 12:51 AM, Paul Baumgart <address@hidden>
>> >> >>> wrote:
>> >> >>>> A different topic this time:
>> >> >>>>
>> >> >>>> The current upload_project() function is a neat feature, but I
>> >> >>>> have
>> >> >>>> two issues with it:
>> >> >>>>
>> >> >>>> 1) It doesn't allow me to exclude certain files/directories (like
>> >> >>>> .git* for example)
>> >> >>>> 2) It uploads the entire project every time, which can be
>> >> >>>> irritating
>> >> >>>> if the change is small but the project is large.
>> >> >>>>
>> >> >>>> So, I made this function, which uses rsync to only upload the
>> >> >>>> differences between the local and remote copies.
>> >> >>>>
>> >> >>>> I would like to get feedback on it, primarily as to whether this
>> >> >>>> would
>> >> >>>> be useful in the mainline Fabric code, and if not, if there is any
>> >> >>>> way
>> >> >>>> to make it be useful.
>> >> >>>>
>> >> >>>> @operation
>> >> >>>> @connects
>> >> >>>> def rsync_project(host, client, env, remotepath, exclude=False,
>> >> >>>> delete=False, extra_opts='', **kwargs):
>> >> >>>> """
>> >> >>>> Uploads the current project directory using rsync, so only
>> >> >>>> changes
>> >> >>>> are
>> >> >>>> uploaded rather than the whole directory like using
>> >> >>>> upload_project.
>> >> >>>>
>> >> >>>> Requires the rsync command-line utility to be available both on
>> >> >>>> the
>> >> >>>> local
>> >> >>>> and the remote machine.
>> >> >>>>
>> >> >>>> Parameters are:
>> >> >>>> remotepath: the path on the remote machine to which
>> >> >>>> to
>> >> >>>> rsync the
>> >> >>>> current project
>> >> >>>> exclude (optional): the string passed to rsync's --exclude
>> >> >>>> option.
>> >> >>>> See the rsync manpage for details.
>> >> >>>> delete (optional): True or False, whether to delete remote
>> >> >>>> files
>> >> >>>> that
>> >> >>>> don't exist locally.
>> >> >>>> extra_opts (optional): Additional command-line options to
>> >> >>>> set
>> >> >>>> for
>> >> >>>> rsync.
>> >> >>>>
>> >> >>>> The rsync command is built from the options as follows:
>> >> >>>> rsync [--delete] [--exclude exclude] -pthrvz [extra_opts]
>> >> >>>> <project dir> <fab_user>@<host>:<remotepath>
>> >> >>>> """
>> >> >>>>
>> >> >>>> remotepath = _lazy_format(remotepath, env)
>> >> >>>>
>> >> >>>> username = ENV.get('fab_user')
>> >> >>>> username = username + '@' if username else username
>> >> >>>>
>> >> >>>> cwd_name = '../' + os.getcwd().split(os.sep)[-1]
>> >> >>>
>> >> >>> This line right here. Why do you go through all this trouble to
>> >> >>> build
>> >> >>> a relative CWD?
>> >> >>
>> >> >> Elementary, dear Watson. To get a prettier output when the assembled
>> >> >> command is printed to the console.
>> >> >>
>> >> >>>
>> >> >>>
>> >> >>>>
>> >> >>>> delete_opt = '--delete' if delete else ''
>> >> >>>>
>> >> >>>> exclude_opt = '--exclude' if exclude else ''
>> >> >>>> exclude = '"' + exclude.strip('"') + '"' if exclude else ''
>> >> >>>>
>> >> >>>> return local('rsync %(delete_opt)s %(exclude_opt)s %(exclude)s
>> >> >>>> -pthrvz %(extra_opts)s %(cwd_name)s
>> >> >>>> %(username)s%(host)s:%(remotepath)s'
>> >> >>>> % locals(), **kwargs) == 0
>> >> >>>>
>> >> >>>>
>> >> >>>> Note that it requires adding a line to the bottom of local():
>> >> >>>>
>> >> >>>> return retcode
>> >> >>>>
>> >> >>>> Regards,
>> >> >>>> Paul
>> >> >>>>
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> Fab-user mailing list
>> >> >>>> address@hidden
>> >> >>>> http://lists.nongnu.org/mailman/listinfo/fab-user
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Venlig hilsen / Kind regards,
>> >> >>> Christian Vest Hansen.
>> >> >>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Venlig hilsen / Kind regards,
>> >> >> Christian Vest Hansen.
>> >> >>
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Venlig hilsen / Kind regards,
>> >> Christian Vest Hansen.
>> >
>> >
>>
>>
>>
>> --
>> Venlig hilsen / Kind regards,
>> Christian Vest Hansen.
>
>
--
Venlig hilsen / Kind regards,
Christian Vest Hansen.