Re: [Fab-user] fabtest

[Mike Korobov]

Hi Jeff,

I also found VirtualBox 4.0.0 and 4.0.2 buggy but with recent 4.0.4
release stability seems to improve a lot. Snapshots taking doesn't
work for me if VM is running in headless mode but in standard mode
(when window pops up and closes) all was fine. This doesn't matter
much because snapshots are taken before tests, during tests snapshots
are only gets restored and restoring works fine for me even in
headless mode.

Snapshots are quite fast: rolling back a snapshot and restoring the
another snapshot takes a few seconds (immediately failing test take
about 2-5 second to run on macbook 2008), this time is small compared
to the time some tests itself take (I'm testing package installation
and pre-configuration on several different linux distributions so it
involve network operations). The nice thing about snapshots is that OS
doesn't have to boot because RAM content is restored too.

I definitely agree this approach is not for everyone and fake ssh
server make sense because it is times faster. Tests against real boxes
should be run only when necessary. Looking forward for your future ssh
testing project!

Fabtest is a small new project (sloccount reports less than 100 lines
of code: VirtualBox api wrapper + fabric utils + a couple of base
TestCases). Not so many tests are written using it now (it's the test
suite for django-fab-deploy), fabtest needs more battle-testing and
some rough edges should be expected, but for existing test suite it
just works.

Didn't know about Vargant, thanks for mentioning it, looks interesting
and polished.

2011/2/22 Jeff Forcier <address@hidden>:
> Hey Mike,
>
> On Sat, Feb 19, 2011 at 5:37 AM, Mike Korobov <address@hidden> wrote:
>
>> I've recently released https://bitbucket.org/kmike/fabtest/ package.
>> This package is for writing tests for Fabric scripts against
>> VirtualBox VMs: before each test VM is rolled back to selected
>> snapshot.
>
> That's neat! I haven't the time to try it out right now but I'm
> curious as to how long your snapshot rollbacks take (1s, 10s, 60s
> etc). I use Virtualbox right now (via Vagrant) but the snapshot
> functionality was somewhat buggy for me so I've not used it much.
> Instead I use Vagrant's package functionality (which is semi slow but
> pretty stable) and/or focus on idempotent Fabric recipes.
>
>> Can please somebody review the code? Are there better ways to do this
>> kind of things? Am I missing something in the implementation?
>
> For running Fabric's own tests, I wrote a somewhat hacky
> Paramiko-driven fake SSH/SFTP server -- it's in the master branch in
> tests/server.py (with to-be-merged improvements to the SFTP stuff in
> put-get-overhaul). It works reasonably well for the existing test
> suite, but needs more work to be generally useful for your use case, I
> think. I have vague plans to clean it up and make it its own project
> sometime.
>
> The greater discussion of "fake SSH server versus real SSH server" is
> one I had a while ago and I'm reasonably happy with the route I took
> for Fabric's test suite, but for "manual" testing of fabfile tasks (as
> opposed to frequent runs of a large test suite) I think your approach
> may have a lot of merit too. Again, I'll try to take a closer look in
> a while.
>
> -Jeff
>
> --
> Jeff Forcier
> Unix sysadmin; Python/Ruby engineer
> http://bitprophet.org
>