8.3 beta, crashed Windows11 VM when trying to snapshot with memory

manilx

Tried to do a snaphot with memory and got this, VM turned stopped cold:

vm.snapshot
{
  "id": "71679fda-1c99-127a-609f-f09edef98abe",
  "saveMemory": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Device_common.QMP_Error(15, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"An IO error has occurred\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000123-15\\\"}\")"
  ],
  "task": {
    "uuid": "f61415bc-7682-e230-d567-2364c4efff5a",
    "name_label": "Async.VM.checkpoint",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20240409T18:10:01Z",
    "finished": "20240409T18:10:04Z",
    "status": "failure",
    "resident_on": "OpaqueRef:476ca210-c084-24b9-7d3d-88df3ebd0bd8",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "xenopsd internal error: Device_common.QMP_Error(15, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"An IO error has occurred\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000123-15\\\"}\")"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/xc/device_common.ml)(line 563))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename ocaml/xenopsd/xc/device.ml)(line 3204))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename ocaml/xenopsd/xc/device.ml)(line 3614))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 1677))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 239))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 244))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 1807))((process xenopsd-xc)(filename result.ml)(line 23))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 1800))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2575))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1828))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1836))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2381))((process xenopsd-xc)(filename list.ml)(line 121))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2374))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2547))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1828))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1836))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3190))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3200))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3220))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 196))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3352))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3913))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3361))((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3367))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 121))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 122))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 143))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 38))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1702))((process xapi)(filename ocaml/xapi/rbac.ml)(line 189))((process xapi)(filename ocaml/xapi/rbac.ml)(line 198))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Device_common.QMP_Error(15, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"An IO error has occurred\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000123-15\\\"}\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Device_common.QMP_Error(15, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"An IO error has occurred\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000123-15\\\"}\"))
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202404091252/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202404091252/packages/xen-api/_getTaskResult.mjs:11:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202404091252/packages/xen-api/index.mjs:1029:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202404091252/packages/xen-api/index.mjs:1063:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202404091252/packages/xen-api/index.mjs:1053:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202404091252/packages/xen-api/index.mjs:1226:14)"
}

Danp

Are you passing through any PCI or USB devices to this VM?

manilx

@Danp None.

Danp

Which version of the VM Tools are you running? I assume the VM has a VTPM, correct?

Does a standard snapshot work? What about a snapshot with memory for a non-Windows 11 VM?

manilx

@Danp Standard snapshot works. Also after rebooting the VM after the crash the snapshot with memory worked.
Tools are: ScreenShot 2024-04-09 at 20.48.28.png

VM has a VTPM.

Danp

Interesting. Maybe @stormi will know more about the reason for the crash.

manilx

@Danp Actually this is not the 1st time this happened as I now remember... Can't recall on which VM though. I 9 out of 10 only do standard snapshots

stormi

Does it always crash, or just sometimes?

manilx

@stormi I don't do this frequently or regularly. Since I reported it crashed 1 out of 4 (tests).

stormi

We'll try to reproduce. If you can upload the tarball created by xen-bugtool -y somewhere private, we can try to have a look at it.

manilx

@stormi said in 8.3 beta, crashed Windows11 VM when trying to snapshot with memory:

xen-bugtool -y

ScreenShot 2024-04-11 at 15.16.20.png
I did run the command on the host it occurred. How do I get the file out?

stormi

With scp (or a windows equivalent such as WinSCP maybe? I seldom use Windows)

manilx

@stormi Worked with sftp. (On macos).

I'll submit link via PM

stormi

Actually, I'm not sure checkpoints are supported, or snapshots of a suspended VM which as a vTPM: https://docs.xenserver.com/en-us/xenserver/8/vms/windows.html#constraints

I had already noted we need to investigate what this actually means, but this might explain why it crashes sometimes.

manilx

@stormi "You cannot snapshot or checkpoint a suspended VM with a vTPM attached."
The VM was running.... And we need snapshots for delta backups.

So let me know what you find.

stormi

@manilx I agree about snapshots. I don't know how we can checkpoint a VM without suspending it (I think it shortly suspends it and them resumes it), so I have trouble understanding XenServer's statement.

jpwg3

I'm experiencing similar issues on 2 out of 8 of my window server 2022 VMs. All 8 VMs with very similar configurations and no vTPM or secure boot on them. The only thing I see in common between the two that fail and the others is all the servers were set with static RAM 8Gb and Dynamic Max 8Gb. Except these two they were set 16 static and 8 dynamic. When I changed them to 8static and 8dynamic it ran just fine.

This isn't a deal breaker for me, but figured it might give you some insight or direction to look.

Xen Orchestra self compile commit ef3ad

stormi

Using dynamic memory is known to be prone to causing occasional issues. Thanks for the feedback.

Oh, by the way, regarding vTPM and snapshots, we finally established that it's fully supported by XenServer 8 and that the documentation was just out of date.