http://wiki.libvirt.org/page/NBD_storage_migration
http://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01573.html


https://bugzilla.redhat.com/show_bug.cgi?id=1201880


About VIR_MIGRATE_TUNNELLED
---------------------------

From libvirt source: src/qemu/qemu_migration.c

/**
 * qemuMigrationDriveMirror:
 * @driver: qemu driver
 * @vm: domain
 * @mig: migration cookie
 * @host: where are we migrating to
 * @speed: how much should the copying be limited
 * @migrate_flags: migrate monitor command flags
 *
 * Run drive-mirror to feed NBD server running on dst and wait
 * till the process switches into another phase where writes go
 * simultaneously to both source and destination. And this switch
 * is what we are waiting for before proceeding with the next
 * disk. On success, update @migrate_flags so we don't tell
 * 'migrate' command to do the very same operation.
 *
 * Returns 0 on success (@migrate_flags updated),
 *        -1 otherwise.
 */


qemu+ssh URI, NBD-based migration without shared storage
--------------------------------------------------------

Complete log here:

    https://kashyapc.fedorapeople.org/virt/temp/nbd-based-libvirt-live-block-migration-with-no-shared-storage-qemu-ssh.log


Selected log snippets below
~~~~~~~~~~~~~~~~~~~~~~~~~~~

2015-04-07 08:46:48.274+0000: 24821: debug : virDomainMigratePerform3Params:5264 : dom=0x7fcab0000d70, (VM: name=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e), dconnuri=qemu+ssh://root@devstack3/system, params=0x7fcab0007920, nparams=0, cookiein=(nil), cookieinlen=0, cookieout=0x7fcad2854b48, cookieoutlen=0x7fcad2854b44, flags=43
2
Complete log here:

    https://kashyapc.fedorapeople.org/virt/temp/nbd-based-libvirt-live-block-migration-with-no-shared-storage-qemu-ssh.log
    
    
    Selected log snippets below
    ~
    Complete log here:
    
        https://kashyapc.fedorapeople.org/virt/temp/nbd-based-libvirt-live-block-migration-with-no-shared-storage-qemu-ssh.log
        
        
        Selected log snippets below
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~
        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    015-04-07 08:46:48.274+0000: 24821: debug : qemuMigrationPerform:4989 : driver=0x7fcac00f9280, conn=0x7fcac02036a0, vm=0x7fcac01c81e0, xmlin=<null>, dconnuri=qemu+ssh://root@devstack3/system, uri=<null>, graphicsuri=<null>, listenAddress=<null>cookiein=<null>, cookieinlen=0, cookieout=0x7fcad2854b48, cookieoutlen=0x7fcad2854b44, flags=43, dname=<null>, resource=0, v3proto=1
.
.
.
2015-04-07 08:46:48.274+0000: 24821: debug : doPeer2PeerMigrate:4699 : driver=0x7fcac00f9280, sconn=0x7fcac02036a0, vm=0x7fcac01c81e0, xmlin=<null>, dconnuri=qemu+ssh://root@devstack3/system, uri=<null>, graphicsuri=<null>, listenAddress=<null>, flags=43, dname=<null>, resource=0
2015-04-07 08:46:48.274+0000: 24821: debug : qemuDomainObjEnterRemote:1687 : Entering remote (vm=0x7fcac01c81e0 name=cvm1)
2015-04-07 08:46:48.274+0000: 24821: debug : virConnectOpenAuth:1339 : name=qemu+ssh://root@devstack3/system, auth=0x7fcac81c89e0, flags=0
2015-04-07 08:46:48.274+0000: 24821: debug : virConnectGetConfigFile:953 : Loading config file '/etc/libvirt/libvirt.conf'
2015-04-07 08:46:48.274+0000: 24821: debug : virConfReadFile:769 : filename=/etc/libvirt/libvirt.conf
2015-04-07 08:46:48.274+0000: 24821: debug : do_open:1132 : name "qemu+ssh://root@devstack3/system" to URI components:
  scheme qemu+ssh
  server devstack3
  user root
  port 0
  path /system
.
.
.
2015-04-07 08:46:48.274+0000: 24821: debug : virCommandRunAsync:2408 : About to run LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin ssh -l root devstack3 sh -c ''\''if '\''nc'\'' -q 2>&1 | grep "requires an argument" >/dev/null 2>&1; then ARG=-q0;else ARG=;fi;'\''nc'\'' $ARG -U /var/run/libvirt/libvirt-sock'\'''
2015-04-07 08:46:48.276+0000: 24821: debug : virCommandRunAsync:2411 : Command result 0, with PID 24800
.
.
.
2015-04-07 08:46:48.497+0000: 24821: debug : doPeer2PeerMigrate3:4415 : driver=0x7fcac00f9280, sconn=0x7fcac02036a0, dconn=0x7fcab0003cb0, dconnuri=qemu+ssh://root@devstack3/system, vm=0x7fcac01c81e0, xmlin=<null>, dname=<null>, uri=<null>, graphicsuri=<null>, listenAddress=<null>, bandwidth=0, useParams=1, flags=43
2015-04-07 08:46:48.497+0000: 24821: debug : qemuMigrationBeginPhase:2615 : driver=0x7fcac00f9280, vm=0x7fcac01c81e0, xmlin=<null>, dname=<null>, cookieout=0x7fcad2854820, cookieoutlen=0x7fcad2854804, flags=43
2015-04-07 08:46:48.497+0000: 24821: debug : qemuDomainObjSetJobPhase:1230 : Setting 'migration out' phase to 'begin3'
2015-04-07 08:46:48.497+0000: 24821: debug : qemuProcessAutoDestroyActive:5725 : vm=cvm1
2015-04-07 08:46:48.497+0000: 24821: debug : virCloseCallbacksGet:197 : vm=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e, conn=(nil)
2015-04-07 08:46:48.497+0000: 24821: debug : virCloseCallbacksGet:207 : cb=(nil)
2015-04-07 08:46:48.497+0000: 24821: debug : qemuMigrationEatCookie:1372 : cookielen=0 cookie='<null>'
.
.
.
2015-04-07 08:46:48.499+0000: 24821: debug : qemuMigrationBakeCookie:1349 : cookielen=293 cookie=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack2</hostname>
  <hostuuid>f87a9d7a-7964-419c-b77f-dd99cd809930</hostuuid>
  <feature name='lockstate'/>
  <nbd>
    <disk target='vda' capacity='41126400'/>
  </nbd>
</qemu-migration>
.
.
.
2015-04-07 08:46:48.499+0000: 24821: debug : qemuDomainDefFormatBuf:1803 : Removing default pci-root from domain 'cvm1' for migration compatibility
2015-04-07 08:46:48.499+0000: 24821: debug : doPeer2PeerMigrate3:4466 : Prepare3 0x7fcab0003cb0
2015-04-07 08:46:48.499+0000: 24821: debug : qemuDomainObjEnterRemote:1687 : Entering remote (vm=0x7fcac01c81e0 name=cvm1)
2015-04-07 08:46:48.774+0000: 24823: debug : virDomainGetJobInfo:8659 : dom=0x7fcab8001500, (VM: name=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e), info=0x7fcad1852af0
2015-04-07 08:46:48.840+0000: 24821: debug : qemuDomainObjExitRemote:1695 : Exited remote (vm=0x7fcac01c81e0 name=cvm1)
2015-04-07 08:46:48.840+0000: 24821: debug : doPeer2PeerMigrate3:4531 : Perform3 0x7fcac02036a0 uri=tcp:devstack3:49152
2015-04-07 08:46:48.840+0000: 24821: debug : qemuDomainObjSetJobPhase:1230 : Setting 'migration out' phase to 'perform3'
2015-04-07 08:46:48.840+0000: 24821: debug : doNativeMigrate:4086 : driver=0x7fcac00f9280, vm=0x7fcac01c81e0, uri=tcp:devstack3:49152, cookiein=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <nbd port='49153'>
    <disk target='vda' capacity='41126912'/>
  </nbd>
</qemu-migration>
, cookieinlen=276, cookieout=0x7fcad2854820, cookieoutlen=0x7fcad2854804, flags=43, resource=0, graphicsuri=<null>
2015-04-07 08:46:48.840+0000: 24821: debug : qemuMigrationRun:3826 : driver=0x7fcac00f9280, vm=0x7fcac01c81e0, cookiein=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <nbd port='49153'>
    <disk target='vda' capacity='41126912'/>
  </nbd>
</qemu-migration>
, cookieinlen=276, cookieout=0x7fcad2854820, cookieoutlen=0x7fcad2854804, flags=43, resource=0, spec=0x7fcad28546d0 (dest=1, fwd=0), dconn=0x7fcab0003cb0, graphicsuri=<null>
2015-04-07 08:46:48.840+0000: 24821: debug : qemuMigrationEatCookie:1372 : cookielen=276 cookie='<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <nbd port='49153'>
    <disk target='vda' capacity='41126912'/>
  </nbd>
</qemu-migration>
'
2015-04-07 08:46:48.841+0000: 24821: debug : qemuMigrationCookieXMLParseStr:1291 : xml=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <nbd port='49153'>
    <disk target='vda' capacity='41126912'/>
  </nbd>
</qemu-migration>


2015-04-07 08:46:48.842+0000: 24821: debug : qemuDomainObjBeginJobInternal:1317 : Starting job: async nested (async=migration out vm=0x7fcac01c81e0 name=cvm1)
2015-04-07 08:46:48.842+0000: 24821: debug : qemuDomainObjBeginJobInternal:1356 : Started job: async nested (async=migration out vm=0x7fcac01c81e0 name=cvm1)
2015-04-07 08:46:48.842+0000: 24821: debug : qemuDomainObjEnterMonitorInternal:1548 : Entering monitor (mon=0x7fcab0010f70 vm=0x7fcac01c81e0 name=cvm1)
2015-04-07 08:46:48.842+0000: 24821: debug : qemuMonitorDriveMirror:3430 : mon=0x7fcab0010f70, device=drive-virtio-disk0, file=nbd:devstack3:49153:exportname=drive-virtio-disk0, format=<null>, bandwidth=8796093022207, granularity=0, buf_size=0, flags=2
2015-04-07 08:46:48.842+0000: 24821: debug : qemuMonitorJSONCommandWithFd:290 : Send command '{"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"nbd:devstack3:49153:exportname=drive-virtio-disk0","speed":8796093022207,"sync":"full","mode":"existing"},"id":"libvirt-11"}' for write with FD -1
2015-04-07 08:46:48.842+0000: 24821: info : qemuMonitorSend:972 : QEMU_MONITOR_SEND_MSG: mon=0x7fcab0010f70 msg={"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"nbd:devstack3:49153:exportname=drive-virtio-disk0","speed":8796093022207,"sync":"full","mode":"existing"},"id":"libvirt-11"}
 fd=-1
2015-04-07 08:46:48.844+0000: 24819: info : qemuMonitorIOWrite:503 : QEMU_MONITOR_IO_WRITE: mon=0x7fcab0010f70 buf={"execute":"drive-mirror","arguments":{"device":"drive-virtio-disk0","target":"nbd:devstack3:49153:exportname=drive-virtio-disk0","speed":8796093022207,"sync":"full","mode":"existing"},"id":"libvirt-11"}
 len=205 ret=205 errno=11
2015-04-07 08:46:48.849+0000: 24819: info : qemuMonitorIOProcess:399 : QEMU_MONITOR_IO_PROCESS: mon=0x7fcab0010f70 buf={"return": {}, "id": "libvirt-11"}
 len=36
.
.
.
2015-04-07 08:46:51.279+0000: 24822: debug : virDomainGetJobInfo:8659 : dom=0x7fcabc001330, (VM: name=cvm1, uuid=ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e), info=0x7fcad2053af0
2015-04-07 08:46:51.350+0000: 24821: debug : qemuMigrationDriveMirror:1794 : Drive mirroring of 'drive-virtio-disk0' completed
.
.
.
2015-04-07 08:46:54.102+0000: 24821: debug : qemuMigrationCancelDriveMirror:1893 : mig=0x7fcab00082d0 nbdPort=0
.
.
.
2015-04-07 08:46:54.104+0000: 24821: debug : qemuMigrationBakeCookie:1349 : cookielen=959 cookie=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack2</hostname>
  <hostuuid>f87a9d7a-7964-419c-b77f-dd99cd809930</hostuuid>
  <nbd>
    <disk target='vda' capacity='41126400'/>
  </nbd>
  <statistics>
    <started>1428396408274</started>
    <stopped>1428396413881</stopped>
    <time_elapsed>5829</time_elapsed>
    <time_remaining>0</time_remaining>
    <downtime>222</downtime>
    <setup_time>1</setup_time>
    <memory_total>1074536448</memory_total>
    <memory_processed>84870527</memory_processed>
    <memory_remaining>0</memory_remaining>
    <memory_bps>32173374</memory_bps>
    <memory_constant>242516</memory_constant>
    <memory_normal>20148</memory_normal>
    <memory_normal_bytes>82526208</memory_normal_bytes>
    <disk_total>0</disk_total>
    <disk_processed>0</disk_processed>
    <disk_remaining>0</disk_remaining>
    <disk_bps>0</disk_bps>
  </statistics>
</qemu-migration>

2015-04-07 08:46:54.104+0000: 24821: debug : qemuDomainObjSetJobPhase:1230 : Setting 'migration out' phase to 'perform3_done'
.
.
.
2015-04-07 08:46:54.112+0000: 24821: debug : doPeer2PeerMigrate3:4622 : Confirm3 0x7fcac02036a0 cancelled=0 vm=0x7fcac01c81e0
2015-04-07 08:46:54.112+0000: 24821: debug : qemuMigrationConfirmPhase:3401 : driver=0x7fcac00f9280, conn=0x7fcac02036a0, vm=0x7fcac01c81e0, cookiein=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <statistics>
    <started>1428396408274</started>
    <stopped>0</stopped>
    <time_elapsed>5259</time_elapsed>
    <time_remaining>0</time_remaining>
    <downtime>222</downtime>
    <setup_time>1</setup_time>
    <memory_total>1074536448</memory_total>
    <memory_processed>84870527</memory_processed>
    <memory_remaining>0</memory_remaining>
    <memory_bps>32173374</memory_bps>
    <memory_constant>242516</memory_constant>
    <memory_normal>20148</memory_normal>
    <memory_normal_bytes>82526208</memory_normal_bytes>
    <disk_total>0</disk_total>
    <disk_processed>0</disk_processed>
    <disk_remaining>0</disk_remaining>
    <disk_bps>0</disk_bps>
  </statistics>
</qemu-migration>
, cookieinlen=885, flags=43, retcode=0
2015-04-07 08:46:54.112+0000: 24821: debug : qemuDomainObjSetJobPhase:1230 : Setting 'migration out' phase to 'confirm3'
2015-04-07 08:46:54.112+0000: 24821: debug : qemuMigrationEatCookie:1372 : cookielen=885 cookie='<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <statistics>
    <started>1428396408274</started>
    <stopped>0</stopped>
    <time_elapsed>5259</time_elapsed>
    <time_remaining>0</time_remaining>
    <downtime>222</downtime>
    <setup_time>1</setup_time>
    <memory_total>1074536448</memory_total>
    <memory_processed>84870527</memory_processed>
    <memory_remaining>0</memory_remaining>
    <memory_bps>32173374</memory_bps>
    <memory_constant>242516</memory_constant>
    <memory_normal>20148</memory_normal>
    <memory_normal_bytes>82526208</memory_normal_bytes>
    <disk_total>0</disk_total>
    <disk_processed>0</disk_processed>
    <disk_remaining>0</disk_remaining>
    <disk_bps>0</disk_bps>
  </statistics>
</qemu-migration>
'
2015-04-07 08:46:54.113+0000: 24821: debug : qemuMigrationCookieXMLParseStr:1291 : xml=<qemu-migration>
  <name>cvm1</name>
  <uuid>ab4c412b-6fdc-4fc4-b78c-f1d49db10d4e</uuid>
  <hostname>devstack3</hostname>
  <hostuuid>5b20d715-c7dc-4101-98f7-a263de6ea200</hostuuid>
  <statistics>
    <started>1428396408274</started>
    <stopped>0</stopped>
    <time_elapsed>5259</time_elapsed>
    <time_remaining>0</time_remaining>
    <downtime>222</downtime>
    <setup_time>1</setup_time>
    <memory_total>1074536448</memory_total>
    <memory_processed>84870527</memory_processed>
    <memory_remaining>0</memory_remaining>
    <memory_bps>32173374</memory_bps>
    <memory_constant>242516</memory_constant>
    <memory_normal>20148</memory_normal>
    <memory_normal_bytes>82526208</memory_normal_bytes>
    <disk_total>0</disk_total>
    <disk_processed>0</disk_processed>
    <disk_remaining>0</disk_remaining>
    <disk_bps>0</disk_bps>
  </statistics>
</qemu-migration>

2015-04-07 08:46:54.113+0000: 24821: debug : qemuProcessStop:5129 : Shutting down vm=0x7fcac01c81e0 name=cvm1 id=31 pid=24605 flags=1
2015-04-07 08:46:54.113+0000: 24821: debug : qemuAgentClose:796 : mon=0x7fcab0004c10
2015-04-07 08:46:54.113+0000: 24821: info : qemuMonitorClose:903 : QEMU_MONITOR_CLOSE: mon=0x7fcab0010f70 refs=2
2015-04-07 08:46:54.114+0000: 24821: debug : qemuProcessKill:5087 : vm=0x7fcac01c81e0 name=cvm1 pid=24605 flags=5
2015-04-07 08:46:54.114+0000: 24821: debug : virProcessKillPainfully:352 : vpid=24605 force=1
2015-04-07 08:46:54.114+0000: 24819: debug : qemuMonitorDispose:258 : mon=0x7fcab0010f70
2015-04-07 08:46:54.114+0000: 24819: debug : qemuAgentDispose:161 : mon=0x7fcab0004c10
2015-04-07 08:46:54.114+0000: 24819: debug : qemuProcessHandleAgentDestroy:190 : Received destroy agent=0x7fcab0004c10 vm=0x7fcac01c81e0


Complete log here:

    https://kashyapc.fedorapeople.org/virt/temp/nbd-based-libvirt-live-block-migration-with-no-shared-storage-qemu-ssh.log