Skip to content

Saving a VM's state fails if it takes over 10 seconds, and the state stays in the disk image #8052

Open

Description

@emkey1

Describe the issue

If saving a VM's state takes over 10 seconds, UTM reports "Timed out waiting for RPC" while QEMU carries on and finishes the save. Nothing records the state, so it stays in the qcow2 with nothing to delete it, using the space of a saved state each time. "Auto save on background" is on by default, so on iOS this can happen unnoticed.

Cause

QEMU runs a snapshot job's work ahead of the answer to the command that started the job, and that work holds up the main loop, so snapshot-save is answered only when the save has ended. QEMUKit waits 10 seconds for any answer (timeoutSeconds in QEMUManager.m). snapshot-load has the same order.

Over QMP, stock QEMU 11.1.2 on macOS 26.5.2, 2 GB of random guest memory:

answered job ended
snapshot-save 2.667 s 2.667 s
snapshot-load 2.207 s 2.207 s

Fix

Put the work behind the answer, in migration/savevm.c: emkey1/qemu@3f572363b8. The commands are then answered in about 1 ms, and QEMUKit's job handling works as its comment describes. utmapp/qemu's utm-edition has the same code. Feel free to take it as it is.

The change
diff --git a/migration/savevm.c b/migration/savevm.c
index 3f8120fa7f..0e400cb79a 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -3828,6 +3828,7 @@ typedef struct SnapshotJob {
     Coroutine *co;
     Error **errp;
     bool ret;
+    QEMUBHFunc *work;
 } SnapshotJob;
 
 static void qmp_snapshot_job_free(SnapshotJob *s)
@@ -3886,13 +3887,30 @@ static void snapshot_delete_job_bh(void *opaque)
     aio_co_wake(s->co);
 }
 
+/*
+ * A snapshot job's work holds up the main loop, so QEMU answers nothing until
+ * it is done, and that includes the command that started the job, whose answer
+ * is still on its way out. A client that gives up on that answer takes a long
+ * save for a failed one, while the save goes on and stays in the image. So the
+ * work goes behind the answer: the monitor's wake-up is waiting in the monitor's
+ * context when this runs there, and the main context's turn comes after it.
+ */
+static void snapshot_job_after_answer_bh(void *opaque)
+{
+    Job *job = opaque;
+    SnapshotJob *s = container_of(job, SnapshotJob, common);
+
+    aio_bh_schedule_oneshot(qemu_get_aio_context(), s->work, job);
+}
+
 static int coroutine_fn snapshot_save_job_run(Job *job, Error **errp)
 {
     SnapshotJob *s = container_of(job, SnapshotJob, common);
     s->errp = errp;
     s->co = qemu_coroutine_self();
-    aio_bh_schedule_oneshot(qemu_get_aio_context(),
-                            snapshot_save_job_bh, job);
+    s->work = snapshot_save_job_bh;
+    aio_bh_schedule_oneshot(iohandler_get_aio_context(),
+                            snapshot_job_after_answer_bh, job);
     qemu_coroutine_yield();
     return s->ret ? 0 : -1;
 }
@@ -3902,8 +3920,9 @@ static int coroutine_fn snapshot_load_job_run(Job *job, Error **errp)
     SnapshotJob *s = container_of(job, SnapshotJob, common);
     s->errp = errp;
     s->co = qemu_coroutine_self();
-    aio_bh_schedule_oneshot(qemu_get_aio_context(),
-                            snapshot_load_job_bh, job);
+    s->work = snapshot_load_job_bh;
+    aio_bh_schedule_oneshot(iohandler_get_aio_context(),
+                            snapshot_job_after_answer_bh, job);
     qemu_coroutine_yield();
     return s->ret ? 0 : -1;
 }
@@ -3913,8 +3932,9 @@ static int coroutine_fn snapshot_delete_job_run(Job *job, Error **errp)
     SnapshotJob *s = container_of(job, SnapshotJob, common);
     s->errp = errp;
     s->co = qemu_coroutine_self();
-    aio_bh_schedule_oneshot(qemu_get_aio_context(),
-                            snapshot_delete_job_bh, job);
+    s->work = snapshot_delete_job_bh;
+    aio_bh_schedule_oneshot(iohandler_get_aio_context(),
+                            snapshot_job_after_answer_bh, job);
     qemu_coroutine_yield();
     return s->ret ? 0 : -1;
 }

How it was tested

An automated test in AOK VM, a fork of UTM with the same QEMUKit (e8e9e87), not in stock UTM, and with its own QEMU.

  • Over QMP on macOS, with the change, x86-64 and ARM64: answered in 1 ms or less, and job times level with the unchanged build when interleaved.
  • iPad Pro 13-inch (M5) simulator, iOS 27.0, x86-64 Alpine with 256 MB of random memory on a qcow2 with 512-byte clusters, to force a save of 34-45 s. Without the change: the error, nothing listed, and a 375 MiB state left in the image, 2 of 2. With it: 5 of 5 saves recorded.
  • Seen without forcing it on an iPad Pro 12.9-inch (2nd generation), iOS 17.7.11: one save in 7, ReactOS at its desktop.

Configuration

  • UTM Version: found in AOK VM; UTM main (a6e0c06) pins the same QEMUKit
  • OS Version: iOS 27.0 (simulator), iOS 17.7.11, macOS 26.5.2
  • Device Model: iPad Pro 13-inch (M5) simulator, iPad Pro 12.9-inch (2nd generation), MacBook Pro (Mac17,6)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions