Windows VMGuest changing network cause the guest to crash

ashinobi

Hi, Everyone.

Good day to everyone, hoping you can help me with my problem.

Problem: I have Win10 installed as a VMGuest, using PV driver 8.2.2.200.rc1. Whenever I change the network in Xen Orchestra (for example from Lab network 3 to lab network 4). The VMGuest crashes, it the VMGuest does not respond to any input. If I access the Statistics tab the VMGuest is on high CPU:

I cannot restart the VMGuest, either using Xen Orchestra or using executing xe vm-reboot.

The error message if I execute a restart in Xen Orchestra is

VM_FAILED_SHUTDOWN_ACKNOWLEDGMENT(OpaqueRef:9e760171-06d1-481c-9660-0ae657631392)

The error message if I execute "xe vm-reboot"

[05:23 home-hypervisor ~]# xe vm-reboot uuid=505fc65d-a9d3-b6de-26bf-87bb7d796e6a
VM didn't acknowledge the need to shutdown.
vm: 505fc65d-a9d3-b6de-26bf-87bb7d796e6a (Lab_Win10_Clnt1)

Using Xen Orchestra, commit 3baa3

I need to know why this is crashing if I try to change the network and how do I restart the VMGuest

olivierlambert

Short term solution to reboot the crashed guest: force reboot.

You might want to try with Citrix PV drivers, to see if you have the same issue (don't forget to clean XCP-ng drivers properly before). We have a new version of the XCP-ng driver coming in the next months, that would be interesting to compare

ashinobi

@olivierlambert

You are right, the issue does not exist when using Citrix drivers. It only happens when using the PV driver.

olivierlambert

Okay so as a workaround, rely on Citrix drivers until we got a new XCP-ng version of the PV drivers

planedrop

I've had something similar in the past and if I recall correctly I had to go into the guest and disable it's network adapter, then make the change on the host, then re-enable the network adapter in the guest VM.

Might be worth a shot to see if that solves it.

I believe with the Citrix drivers though it doesn't happen.

piotrlotr1

Apologize again for touching an old topic but we are just testing all things we need from XCP-ng while migrating OFF VMware and it looks like this error still occurs even with Citrix XenTools installed.

Version we have inside VMs is 9.4.0-146.

Here is the result of removing network adapter.

Not every time I delete network adapter the issue occur.

We've tried on different VMs with Windows 2019 and the problem exists.

XCP-ng-JustGreat

All, Just to let you know that I have observed this same behavior on Windows 11 24H2 with the latest Citrix drivers. After changing the network, the CPU spikes to 100% and the VM becomes unusable. In that case, the only fix is to perform a forced reboot. As others have noted here, sometimes the VM will continue to run okay, but the CPU spike failure happens more often than not.

piotrlotr1

@XCP-ng-JustGreat
Check mentioned by @planedrop disabling network adapter from devmgmt.msc before removing it from XOA. It works in our case as a workaround but of course the problem is still there.

XCP-ng-JustGreat

@piotrlotr1 Will definitely use the trick if needed in the future. Normally, changing the network for a running VM is not something we typically need to do. Thanks!

piotrlotr1

There is one more thing we've noticed.

While CPU are 100% flat, when we trigger migration to different XCP-ng host the task stops at 90% and goes "failure".

Here's the code.

{
  "id": "0m9005wjh",
  "properties": {
    "method": "vm.migrate",
    "params": {
      "vm": "c30f10de-4dfa-e7dd-84dd-3104ed7e1c23",
      "migrationNetwork": "6dc5b9a9-36dc-ab29-4ad6-80ecbd53b906",
      "targetHost": "24eacfb2-da63-4e27-aa02-6e5cb774026b"
    },
    "name": "API call: vm.migrate",
    "userId": "612f2ca4-dddf-4d18-96c4-1ce8c57a6383",
    "type": "api.call"
  },
  "start": 1743602926589,
  "status": "failure",
  "updatedAt": 1743602977282,
  "end": 1743602977282,
  "result": {
    "code": "INTERNAL_ERROR",
    "params": [
      "Xenops_interface.Xenopsd_error(S(Failed_to_acknowledge_suspend_request))"
    ],
    "task": {
      "uuid": "2f0c5622-52a1-f621-bbec-b2cdf1ab07c9",
      "name_label": "Async.VM.migrate_send",
      "name_description": "",
      "allowed_operations": [],
      "current_operations": {},
      "created": "20250402T14:08:46Z",
      "finished": "20250402T14:09:37Z",
      "status": "failure",
      "resident_on": "OpaqueRef:d11c5a86-fa0b-7788-3f80-0cb300322d88",
      "progress": 1,
      "type": "<none/>",
      "result": "",
      "error_info": [
        "INTERNAL_ERROR",
        "Xenops_interface.Xenopsd_error(S(Failed_to_acknowledge_suspend_request))"
      ],
      "other_config": {},
      "subtask_of": "OpaqueRef:NULL",
      "subtasks": [],
      "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2561))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 1706))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 239))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 244))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 1840))((process xenopsd-xc)(filename result.ml)(line 23))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 1833))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2554))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1830))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1838))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2380))((process xenopsd-xc)(filename list.ml)(line 121))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2373))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2746))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 403))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2759))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 403))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2657))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1830))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 1838))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3185))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3195))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3215))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 194))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3387))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3555))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 264))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 270))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 295))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1551))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1714))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 143))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2591))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
    },
    "message": "INTERNAL_ERROR(Xenops_interface.Xenopsd_error(S(Failed_to_acknowledge_suspend_request)))",
    "name": "XapiError",
    "stack": "XapiError: INTERNAL_ERROR(Xenops_interface.Xenopsd_error(S(Failed_to_acknowledge_suspend_request)))\n    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)\n    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1072:24)\n    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1106:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1096:12)\n    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1269:14)"
  }
}

Maybe Devs will get something out of it : - /

olivierlambert

This is because the VM is likely too busy to migrate memory pages fast enough vs what's newly written.

olivierlambert

Adding @dinhngtu in the loop

dinhngtu

This is a driver bug that we fixed in XCP-ng Windows tools v9.0.9030 but hasn't been integrated by Citrix yet. You can try it out if you're not running a production system.

olivierlambert

@dinhngtu said in Windows VMGuest changing network cause the guest to crash:

This is a driver bug that we fixed in XCP-ng Windows tools v9.0.9030 but hasn't been integrated by Citrix yet. You can try it out if you're not running a production system.

Gosh, we are now officially faster than Citrix to detect and fix bugs even in the Windows PV driver