[Qemu-commits] [qemu/qemu] 42da55: migration: set state to post-migrate on failure

[GitHub]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 42da5550d6d44ea0a5e36925deba1e1b13041f42
      
https://github.com/qemu/qemu/commit/42da5550d6d44ea0a5e36925deba1e1b13041f42
  Author: Dr. David Alan Gilbert <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    M migration/migration.c

  Log Message:
  -----------
  migration: set state to post-migrate on failure

If a migration fails/is cancelled during the postcopy stage we currently
end up with the runstate as finish-migrate, where it should be post-migrate.
There's a small window in precopy where I think the same thing can
happen, but I've never seen it.

It rarely matters; the only postcopy case is if you restart a migration, which
again is a case that rarely matters in postcopy because it's only
safe to restart the migration if you know the destination hasn't
been running (which you might if you started the destination with -S
and hadn't got around to 'c' ing it before the postcopy failed).
Even then it's a small window but potentially you could hit if
there's a problem loading the devices on the destination.

This corresponds to:
https://bugzilla.redhat.com/show_bug.cgi?id=1355683

Signed-off-by: Dr. David Alan Gilbert <address@hidden>
Reviewed-by: Amit Shah <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 6f7a4a81ce6cc75a1b3b8f4dddaedf26dc04b705
      
https://github.com/qemu/qemu/commit/6f7a4a81ce6cc75a1b3b8f4dddaedf26dc04b705
  Author: Daniel P. Berrange <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    A scripts/qmp/__init__.py

  Log Message:
  -----------
  scripts: add __init__.py file to scripts/qmp/

When searching for modules to load, python will ignore any
sub-directory which does not contain __init__.py. This means
that both scripts and scripts/qmp/ have to be explicitly added
to the python path. By adding a __init__.py file to scripts/qmp,
we only need add scripts/ to the python path and can then simply
do 'from qmp import qmp' to load scripts/qmp/qmp.py.

Signed-off-by: Daniel P. Berrange <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 991e7c46504807bd89ba8debeccc5211e0b7f221
      
https://github.com/qemu/qemu/commit/991e7c46504807bd89ba8debeccc5211e0b7f221
  Author: Daniel P. Berrange <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    M scripts/qmp/qmp.py

  Log Message:
  -----------
  scripts: add a 'debug' parameter to QEMUMonitorProtocol

Add a 'debug' parameter to the QEMUMonitorProtocol class
which will cause it to print out all JSON strings on
sys.stderr

Signed-off-by: Daniel P. Berrange <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 66613974468fb6e1609fb3eabf55981b1ee436cf
      
https://github.com/qemu/qemu/commit/66613974468fb6e1609fb3eabf55981b1ee436cf
  Author: Daniel P. Berrange <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    A scripts/qemu.py
    M scripts/qtest.py
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  scripts: refactor the VM class in iotests for reuse

The iotests module has a python class for controlling QEMU
processes. Pull the generic functionality out of this file
and create a scripts/qemu.py module containing a QEMUMachine
class. Put the QTest integration support into a subclass
QEMUQtestMachine.

Signed-off-by: Daniel P. Berrange <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 238064621fcf5fabedc36d2acdfffa72c723a748
      
https://github.com/qemu/qemu/commit/238064621fcf5fabedc36d2acdfffa72c723a748
  Author: Daniel P. Berrange <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    M scripts/qmp/qmp.py

  Log Message:
  -----------
  scripts: set timeout when waiting for qemu monitor connection

If QEMU fails to launch for some reason, the QEMUMonitorProtocol
class accept() method will wait forever in a socket accept call.
Set a timeout of 15 seconds so that we fail more gracefully
instead of hanging the test script forever

Signed-off-by: Daniel P. Berrange <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 168ae6c24bf1cfc978f769292705229af82818e8
      
https://github.com/qemu/qemu/commit/168ae6c24bf1cfc978f769292705229af82818e8
  Author: Daniel P. Berrange <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    M scripts/qmp/qmp.py

  Log Message:
  -----------
  scripts: ensure monitor socket has SO_REUSEADDR set

If tests use a TCP based monitor socket, the connection will
go into a TIMED_WAIT state when the test exits. This will
randomly prevent the test from being re-run without a certain
time period. Set the SO_REUSEADDR flag on the socket to ensure
we can immediately re-run the tests

Signed-off-by: Daniel P. Berrange <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: 409437e16df273fc5f78f6cd1cb53023eaeb9b72
      
https://github.com/qemu/qemu/commit/409437e16df273fc5f78f6cd1cb53023eaeb9b72
  Author: Daniel P. Berrange <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    M configure
    M tests/Makefile.include
    A tests/migration/.gitignore
    A tests/migration/guestperf-batch.py
    A tests/migration/guestperf-plot.py
    A tests/migration/guestperf.py
    A tests/migration/guestperf/__init__.py
    A tests/migration/guestperf/comparison.py
    A tests/migration/guestperf/engine.py
    A tests/migration/guestperf/hardware.py
    A tests/migration/guestperf/plot.py
    A tests/migration/guestperf/progress.py
    A tests/migration/guestperf/report.py
    A tests/migration/guestperf/scenario.py
    A tests/migration/guestperf/shell.py
    A tests/migration/guestperf/timings.py
    A tests/migration/stress.c

  Log Message:
  -----------
  tests: introduce a framework for testing migration performance

This introduces a moderately general purpose framework for
testing performance of migration.

The initial guest workload is provided by the included 'stress'
program, which is configured to spawn one thread per guest CPU
and run a maximally memory intensive workload. It will loop
over GB of memory, xor'ing each byte with data from a 4k array
of random bytes. This ensures heavy read and write load across
all of guest memory to stress the migration performance. While
running the 'stress' program will record how long it takes to
xor each GB of memory and print this data for later reporting.

The test engine will spawn a pair of QEMU processes, either on
the same host, or with the target on a remote host via ssh,
using the host kernel and a custom initrd built with 'stress'
as the /init binary. Kernel command line args are set to ensure
a fast kernel boot time (< 1 second) between launching QEMU and
the stress program starting execution.

None the less, the test engine will initially wait N seconds for
the guest workload to stablize, before starting the migration
operation. When migration is running, the engine will use pause,
post-copy, autoconverge, xbzrle compression and multithread
compression features, as well as downtime & bandwidth tuning
to encourage completion. If migration completes, the test engine
will wait N seconds again for the guest workooad to stablize on
the target host. If migration does not complete after a preset
number of iterations, it will be aborted.

While the QEMU process is running on the source host, the test
engine will sample the host CPU usage of QEMU as a whole, and
each vCPU thread. While migration is running, it will record
all the stats reported by 'query-migration'. Finally, it will
capture the output of the stress program running in the guest.

All the data produced from a single test execution is recorded
in a structured JSON file. A separate program is then able to
create interactive charts using the "plotly" python + javascript
libraries, showing the characteristics of the migration.

The data output provides visualization of the effect on guest
vCPU workloads from the migration process, the corresponding
vCPU utilization on the host, and the overall CPU hit from
QEMU on the host. This is correlated from statistics from the
migration process, such as downtime, vCPU throttling and iteration
number.

While the tests can be run individually with arbitrary parameters,
there is also a facility for producing batch reports for a number
of pre-defined scenarios / comparisons, in order to be able to
get standardized results across different hardware configurations
(eg TCP vs RDMA, or comparing different VCPU counts / memory
sizes, etc).

To use this, first you must build the initrd image

 $ make tests/migration/initrd-stress.img

To run a a one-shot test with all default parameters

 $ ./tests/migration/guestperf.py > result.json

This has many command line args for varying its behaviour.
For example, to increase the RAM size and CPU count and
bind it to specific host NUMA nodes

 $ ./tests/migration/guestperf.py \
       --mem 4 --cpus 2 \
       --src-mem-bind 0 --src-cpu-bind 0,1 \
       --dst-mem-bind 1 --dst-cpu-bind 2,3 \
       > result.json

Using mem + cpu binding is strongly recommended on NUMA
machines, otherwise the guest performance results will
vary wildly between runs of the test due to lucky/unlucky
NUMA placement, making sensible data analysis impossible.

To make it run across separate hosts:

 $ ./tests/migration/guestperf.py \
       --dst-host somehostname > result.json

To request that post-copy is enabled, with switchover
after 5 iterations

 $ ./tests/migration/guestperf.py \
       --post-copy --post-copy-iters 5 > result.json

Once a result.json file is created, a graph of the data
can be generated, showing guest workload performance per
thread and the migration iteration points:

 $ ./tests/migration/guestperf-plot.py --output result.html \
  --migration-iters --split-guest-cpu result.json

To further include host vCPU utilization and overall QEMU
utilization

 $ ./tests/migration/guestperf-plot.py --output result.html \
  --migration-iters --split-guest-cpu \
        --qemu-cpu --vcpu-cpu result.json

NB, the 'guestperf-plot.py' command requires that you have
the plotly python library installed. eg you must do

 $ pip install --user  plotly

Viewing the result.html file requires that you have the
plotly.min.js file in the same directory as the HTML
output. This js file is installed as part of the plotly
python library, so can be found in

  $HOME/.local/lib/python2.7/site-packages/plotly/offline/plotly.min.js

The guestperf-plot.py program can accept multiple json files
to plot, enabling results from different configurations to
be compared.

Finally, to run the entire standardized set of comparisons

  $ ./tests/migration/guestperf-batch.py \
       --dst-host somehost \
       --mem 4 --cpus 2 \
       --src-mem-bind 0 --src-cpu-bind 0,1 \
       --dst-mem-bind 1 --dst-cpu-bind 2,3
       --output tcp-somehost-4gb-2cpu

will store JSON files from all scenarios in the directory
named tcp-somehost-4gb-2cpu

Signed-off-by: Daniel P. Berrange <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Amit Shah <address@hidden>


  Commit: e3643d32ee805d8a7b248526a507452c484a98e5
      
https://github.com/qemu/qemu/commit/e3643d32ee805d8a7b248526a507452c484a98e5
  Author: Peter Maydell <address@hidden>
  Date:   2016-07-22 (Fri, 22 Jul 2016)

  Changed paths:
    M configure
    M migration/migration.c
    A scripts/qemu.py
    A scripts/qmp/__init__.py
    M scripts/qmp/qmp.py
    M scripts/qtest.py
    M tests/Makefile.include
    A tests/migration/.gitignore
    A tests/migration/guestperf-batch.py
    A tests/migration/guestperf-plot.py
    A tests/migration/guestperf.py
    A tests/migration/guestperf/__init__.py
    A tests/migration/guestperf/comparison.py
    A tests/migration/guestperf/engine.py
    A tests/migration/guestperf/hardware.py
    A tests/migration/guestperf/plot.py
    A tests/migration/guestperf/progress.py
    A tests/migration/guestperf/report.py
    A tests/migration/guestperf/scenario.py
    A tests/migration/guestperf/shell.py
    A tests/migration/guestperf/timings.py
    A tests/migration/stress.c
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  Merge remote-tracking branch 
'remotes/amit-migration/tags/migration-for-2.7-6' into staging

Migration:
- Fix a postcopy bug
- Add a testsuite for measuring migration performance

# gpg: Signature made Fri 22 Jul 2016 08:56:44 BST
# gpg:                using RSA key 0xEB0B4DFC657EF670
# gpg: Good signature from "Amit Shah <address@hidden>"
# gpg:                 aka "Amit Shah <address@hidden>"
# gpg:                 aka "Amit Shah <address@hidden>"
# Primary key fingerprint: 48CA 3722 5FE7 F4A8 B337  2735 1E9A 3B5F 8540 83B6
#      Subkey fingerprint: CC63 D332 AB8F 4617 4529  6534 EB0B 4DFC 657E F670

* remotes/amit-migration/tags/migration-for-2.7-6:
  tests: introduce a framework for testing migration performance
  scripts: ensure monitor socket has SO_REUSEADDR set
  scripts: set timeout when waiting for qemu monitor connection
  scripts: refactor the VM class in iotests for reuse
  scripts: add a 'debug' parameter to QEMUMonitorProtocol
  scripts: add __init__.py file to scripts/qmp/
  migration: set state to post-migrate on failure

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/206d0c24361a...e3643d32ee80