"Hardware-assisted virtualization is not enabled on this host" even though platform:exp-nested-hvm=true is set

abudef

Nested virtualization seems to work correctly in version 8.2.1 with the latest updates (~~native host~~).

On the contrary; the problem is related to virtualized XCP-ng version 8.3 with the latest updates, not to the native host.

abudef

Nested virtualization seems to work correctly in version 8.2.1 with the latest updates (~~native host~~).

On the contrary; the problem is related to virtualized XCP-ng version 8.3 with the latest updates, not to the native host.

The combination (8.3latest(8.x)) does not work either. Nested virtualization simply doesn't work if 8.3 is either a native host, a virtualized host, or both.

olivierlambert

And 8.3 with Xen 4.13 instead of 4.17?

abudef

Well, it seems that the problem is caused by Xen 4.17...

abudef

I temporarily moved the entire XCP-ng lab to ESXi. I need a few different hosts for scenario testing, migration and backup testing, XO proxies testing etc. and of course I can't dedicate a hw server for each of them in the lab. Please let me know when the nested virtualization support will be operational again on version 8.3.

olivierlambert

Well, you are the first to report this regression, so we'll need time to reproduce and investigate.

olivierlambert

Add @stormi so we create a card internally.

stormi

Noted! If this happens on our hardware too, this will be very annoying for our own virtual XCP-ng that we use all the time for development and testing.

abudef

@olivierlambert Thank you very much for the impeccable support and fast communication

I tested again today and the problem is always on the host side, regardless of the type of guest. This corrects some of my hasty information above.

stormi

@abudef said in "Hardware-assisted virtualization is not enabled on this host" even though platform:exp-nested-hvm=true is set:

I tested again today and the problem is always on the host side, regardless of the type of guest. This corrects some of my hasty information above.

What do you mean? Can you give examples?

abudef

@stormi If the host is on version 8.3 with Xen 4.17, nested virtualization does not work on any guest. If version 8.3 with Xen 4.17 is on a guest, but the host is running version 8.2.1 or an out-of-date 8.3 (Xen 4.13), it doesn't matter and nested guest virtualization works.

stormi

@abudef In the second case, do you mean there's:

8.2.1 host running a nested 8.3 (Xen 4.17) VM running itself a Nested XCP-ng? (so nested inside nested)

abudef

@stormi That's not quite what I meant. The point is that if the host is 8.3/4.17, then virtualization is not working on its guest and therefore no other nested guest can be started on this guest. And actually, yes, if 8.3/4.17 appears somewhere in the cascade of nested hosts, then nested virtualization will no longer work on its guest.

olivierlambert

@abudef Can you try this on a shutdown VM:

Disable nested from the VM/Advanced tab
xe vm-param-set platform:nested_virt=true uuid=<VM UUID>
Start the VM

olivierlambert

It seems we also need to modify some other bits, stand by.

abudef

Disable nested from the VM/Advanced tab
xe vm-param-set platform:nested_virt=true uuid=<VM UUID>
Start the VM

...unfortunately it didn't work, I assume because...

It seems we also need to modify some other bits

Andrew

@olivierlambert On XCP 8.3/Xen 4.17 the whole vmx option is missing from any VM's CPU.... Waiting for the other bits...

olivierlambert

Yes, that's exactly the missing piece, but it has to be computed from the hex string on the pool platform CPU.

abudef

As expected, the same problem is with XenServer 8/4.17

abudef

@olivierlambert @stormi Please do you have any idea when this problem might be resolved? The question is how to deal with the test lab, whether to wait, because a secondary problem is that the nested VMs cannot be migrated elsewhere from the affected virtualized XCP-ng hosts.

vm.migrate
{
  "vm": "654cc5c6-7e50-fc28-ecc4-fe46929905b2",
  "mapVifsNetworks": {
    "2db4235a-345f-f286-4172-77dab4e198fe

": "8e969c1a-cafa-7ac0-504d-cf5cd19ef1e4

"
  },
  "migrationNetwork": "8e969c1a-cafa-7ac0-504d-cf5cd19ef1e4

",
  "sr": "a25ba333-a1a5-f22f-c337-0ec662e835ed",
  "targetHost": "ca60fce7-924a-45f9-a1c6-ee860952e6aa"
}
{
  "code": "NO_HOSTS_AVAILABLE",
  "params": [],
  "task": {
    "uuid": "57fe6efb-569e-3fa3-a345-d43377260884

",
    "name_label": "Async.VM.migrate_send",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20240510T10:45:50Z",
    "finished": "20240510T10:45:51Z",
    "status": "failure",
    "resident_on": "OpaqueRef:99de1bb8-8e7f-e79a-d7f9-5d84c2c09a73",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "NO_HOSTS_AVAILABLE"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 106))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1453))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1506))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1445))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2537))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2559))((process xapi)(filename ocaml/xapi/rbac.ml)(line 189))((process xapi)(filename ocaml/xapi/rbac.ml)(line 198))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
  },
  "message": "NO_HOSTS_AVAILABLE()",
  "name": "XapiError",
  "stack": "XapiError: NO_HOSTS_AVAILABLE()
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202405091612/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202405091612/packages/xen-api/_getTaskResult.mjs:11:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202405091612/packages/xen-api/index.mjs:1029:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202405091612/packages/xen-api/index.mjs:1063:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202405091612/packages/xen-api/index.mjs:1053:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202405091612/packages/xen-api/index.mjs:1226:14)"
}

(targetHost above is a common native XCP-ng host)