VM Console Screen Suddenly Inaccessible
-
Good-day Folks,
I trust you're all doing well. For the past couple of days, I've been noticing that the Console of one of my VMs (DC02) becomes inaccessible (from within Xen Orchestra), however, during this state I am able to using RDP to remotely access the VM.
To begin troubleshooting, I attempted to reboot the VM from Xen Orchestra's VM controls menus. This failed and I was greeted by the following error message:
INTERNAL_ERROR: Object with type VM and id 9fa84df4-3912-5cbf-09a6-3374dd27eead/config does not exist in xenopsd. Next, I attempted to force a VM Reboot or Shutdown from the VM's Advanced tab, and was met with the same error message.The Temporary Solution (Workaround)
This is what got my VM back into a working/running state - though I'm not sure if the order is important:- First, I ran
Rescan all diskson the SR where the VDI of the VM was located - Second, I restarted the Toolstack of the host that the VM was running on. I immediately noticed that the previous shutdown attempt took effect, but I was now unable to restart the VM. All attempts to start the VM resulted in the error:
INTERNAL_ERROR: xenopsd internal error: Storage_error ([S(Illegal_transition);[[S(Attached);S(RO)];[S(Attached);S(RW)]]]) - Last, I restarted the Toolstack of the master host. Once Xen Orchestra reconnected, I was able to start up the VM without any issues.
Unfortunately, I am unable to tell when the issue is occurring or what conditions lead to the VM getting into this state. Since this is a lab environment being used for a Proof of Concept (POC), we're in the lab sporadically. I've observed this issue twice now, generally in the mornings.
Anyway, I thought I'd report it to the community to see if anyone has encountered a similar issue before and could offer some hints on a permanent solution. Thanks.
My Environment:
Hosts: 3-node XCP-ng v8.3.0
XOA: v5.102.1 Build: 20250220 (air-gapped)
- First, I ran
-
Hmm my gut feeling: a failed migration leaving a "ghost" of the original VM
-
@olivierlambert said in VM Console Screen Suddenly Inaccessible:
Hmm my gut feeling: a failed migration leaving a "ghost" of the original VM
I agree, I've had this happen a few times myself and it's always something with a failed migration (using local storage for a VM during a rolling pool update for example).
@kagbasi-ngc what work occurred on the pool ahead of this issue? Though you've rebooted the entire environment so I don't think anything will be in the XO logs any more, unless you're using a log shipper of some sort to keep these somewhere else.
-
Hmm, interesting observation about potentially this being a failed migration issue.
I can confirm that none of my users in the lab are migrating any VMs or this particular one, for that matter. However, I do have the Load Balancer plugin running and it could be the source of a migration that could've failed. How can I confirm this in the logs? I mean, what should I be looking for?
-
@kagbasi-ngc The support team may know better than I do,
dmesglogs may have something but I'm not 100% on where migration information is logged.... -
@kagbasi-ngc said in VM Console Screen Suddenly Inaccessible:
Hmm, interesting observation about potentially this being a failed migration issue.
I can confirm that none of my users in the lab are migrating any VMs or this particular one, for that matter. However, I do have the Load Balancer plugin running and it could be the source of a migration that could've failed. How can I confirm this in the logs? I mean, what should I be looking for?
While trying to find logs, you can try to find the VM in question and see if it is using local storage rather than shared.
