Bug --- https://bugs.launchpad.net/nova/+bug/1334398 libvirt live_snapshot periodically explodes on libvirt 1.2.2 in the gate What's the next step to debug ----------------------------- Attach GDB to the hung QEMU process in the OpenStack gate infrastructure to get stack traces. NOTE: it'd be very useful to have an env variable that can be toggled to enable libvirt/QEMU run under `gdb` for $REVIEW. Context and analysis -------------------- This bug[1] has received some debugging by DanB (analysis in the bug) and I've also seen some amount of IRC debugging between DanB & Eric Blake on #virt, OFTC, during JUL-2014 (summary, here[*]. Version: Ubuntu 14.04 LTS with QEMU 2.0 and libvirt 1.2.2 The bug has a lot of comments, posting some relevant context below. Cause of failure, comment#17 from the bug[2] (danpb), and relevant Nova libvirt code here[3]. Nova uses libvirt blockRebase Python API as below: "The actual one that is failing is in this code [. . .] # NOTE (rmk): Establish a temporary mirror of our root disk and # issue an abort once we have a complete copy. domain.blockRebase(disk_path, disk_delta, 0, libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY | libvirt.VIR_DOMAIN_BLOCK_REBASE_REUSE_EXT | libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW) while self._wait_for_block_job(domain, disk_path): time.sleep(0.5) domain.blockJobAbort(disk_path, 0) [. . .] So we've done a block rebase, and then when we finish waiting for it to finish, we abort the job and at that point we see the crashed QEMU. The QEMU logs are stuff something useful to get" I posted a sequence of calls virDomain* libvirt calls that are being made in comment#23[4] You can notice that Dan has captured a failure in comment #27 of the bug[5]: where it issues a QEMU monitor command 'query-block-jobs': 2014-07-09 11:05:50.705+0000: 21774: debug : qemuMonitorJSONCommandWithFd:264 : Send command '{"execute":"query-block-jobs","id":"libvirt-15"}' fo +r write with FD -1 and, not getting any reply. "In the time where libvirtd is waiting for the QEMU response, there are many other libvirt API calls taking place, so it doesn't look like libvirtd itself is misbehaving. It also successfully sees the EOF on the QEMU monitor socket, so we can see the event loop is still listening for the reply from QEMU. So I'm still left thinking this is a QEMU bug of some sort." [1] https://bugs.launchpad.net/nova/+bug/1334398 -- libvirt live_snapshot periodically explodes on libvirt 1.2.2 in the gate [2] https://bugs.launchpad.net/nova/+bug/1334398/comments/17 [3] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1802 [4] https://bugs.launchpad.net/nova/+bug/1334398/comments/23 the sequence that was performed in the libvirt bug: 1. virDomainGetXMLDesc 2. virDomainBlockJobAbort 3. virDomainUndefine 4. virDomainBlockRebase 5. virDomainBlockJobAbort 6. virDomainDefineXML [5] https://bugs.launchpad.net/nova/+bug/1334398/comments/27 [6] Most recent discussion of this bug: http://lists.openstack.org/pipermail/openstack-dev/2014-December/052469.html -- bug 1334398 and libvirt live snapshot support [*] #virt, IRC discussion between danpb and eblake, 9JUL2014: ----------------------------------------------------------------- : i finally captured a libvirtd.log file - see comment #27 in that bug, which shows it sending msg={"execute":"query-block-jobs","id":"libvirt-15"} and never getting a reply : so I'm thinking the QEMU query-block-jobs method has some hang / infinite loop in it : QEMU block layer is multi threaded now right ? : stefanha has done some work to make the dataplane multithreaded, but it was experimental in 2.0 and had to be requested (libvirt did not request it). so for now, you can still assume the block layer is single-threaded, with coroutines allowing for parallel operations to compete for event loop attention : my current thinking is that qmp_query_block_jobs() is iterating over the BlockDriverState list global variable, while the drive mirror job is in the process of deleting the BlockDriveState for the mirror and thus ends up in an infinite loop, that's the only way I can see for qmp_query_block_jobs() to hang indefinitely : okay, looking at blockdev.c now; qmp_query_block_jobs() just does a bdrv_iterate, with a callback function that adds to the list for each BDS that has a job attached to it : so if there's a bug, it would be in bdrv_iterate() : i don't see any mutex locking anywhere in the functions accessing that list : so they must be assuming some thread serialization in the callers : so i'm wondering if something is breaking that assumption unless it is protected by the BQL : that could also be true - I don't see anything preventing bs->job from changing from non-NULL to NULL unless it is protected by the BQL : I think bs->job is not changing from non-NULL to NULL, because block_job_query would segfault, but you are seeing an infinite loop : is there a way to attach gdb to the hung qemu process when you detect that a response isn't happening fast enough?