@poddingue

Okay, we're back here. I was told I have a hung CIFS mount that no longer exists, so a host needs to reboot to clear it, and faulty XOSTOR resources. But I can't reboot the host until I can move VMs off. Is there any indication of how to repair the faulty XOSTOR resources so I can move the VMs and reboot the host?

Thanks for taking a look.

@Danp

I'm US Central time, so I'll open the tunnel around 2:00am tomorrow. Who do I send the code to?

Thanks!

@Danp

I have my XOA registered, so let me know how I go about getting you into it.

@Danp

New VMs just hang with the XCP-ng rocketship logo in the console. I've tried UEFI and traditional BIOS.

@Danp

╭───────────────────────────────────────────────────────────────────╮
┊ Node               ┊ NodeType ┊ Addresses                ┊ State  ┊
╞═══════════════════════════════════════════════════════════════════╡
┊ host1  ┊ COMBINED ┊ XX.XX.XX.16:3366 (PLAIN) ┊ Online ┊
┊ host2  ┊ COMBINED ┊ XX.XX.XX.17:3366 (PLAIN) ┊ Online ┊
┊ host3  ┊ COMBINED ┊ XX.XX.XX.15:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────────────╯
[16:14 host1 ~]# linstor r l --faulty
╭──────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞══════════════════════════════════════════════════════════════════╡
╰──────────────────────────────────────────────────────────────────╯

@poddingue

sm-3.2.12-23.5.xcpng8.3.x86_64 across all three hosts.

Just tried running through the chain of repair and restart, and I'm getting this on the latest as well:

[12:49 host1 ~]# xe vm-start uuid=c0faf508-9f14-e091-43b1-d6c5ef7e07d4
Error code: SR_BACKEND_FAILURE_1200
Error parameters: , [Errno 30] Read-only file system: '/dev/drbd/by-res/xcp-volume-693829e3-0b60-4b0c-bed2-611995ec83f1/0',

I am having an issue with my XCP cluster, running Linstor across three nodes.

Migrations don't complete. The final handoff doesn't finish, and they sit at 100%. VM reboots result in the underlying VDI becoming unavailable. A VM that was running just fine gets a reboot and I have to go through the process of 'xl list' 'xl destroy <X>' etc. And then on attempted start:

Error code: SR_BACKEND_FAILURE_46
Error parameters: , The VDI is not available [opterr=VDI 6c8fe996-2466-47ef-993b-0de3b66da92e not detached cleanly],

I can't snapshot or do full backups. The task just hangs.
Edit: I just attempted a snapshot of a VM via the official XO Appliance, and got this:

SR_BACKEND_FAILURE_82(, Failed to snapshot VDI [opterr=['MAP_DUPLICATE_KEY', 'VDI', 'sm_config', 'OpaqueRef:cbf6441d-c558-725b-8b21-39609026f507', 'paused']], )

This is becoming a regular occurrence, and I'm not sure what to do to stop this. My environment is slowly degrading. But it definitely seems like its an issue within the SR.

Any help would be appreciated.

@acebmxer

Yeah, it does not show up as an available device in XOA or XCP-ng Center. Just listed when running lspci

At one point there was a 10GbE NIC in one of my servers, but has since been removed. A GPU has been installed, but the system is showing the old nic still sitting at 07:00.0 which is also the GPU.

What would be the best way to get rid of this ghost nic from the system so that I can effectively get this GPU assigned?

Thanks!

I've got my OPNsense (FreeBSD) router running in a VM on XCP, and it does have the xe-guest-utilities installed. I'm trying to figure out an issue on one of the interfaces, and I noticed a weird discrepancy in XO with this VM. The MAC addresses and such for the interfaces being used by the VM are all off.

It seems that VIF #0 is the only one coming up correctly.

Unfortunately, I don't think the FreeBSD or OPNsense teams have a maintainer for the xe-guest-utilities package, and so what they distribute is showing as 6.2.1-76888 in XO.

I don't know if this is what is causing my network issue, but I thought it strange.

@fohdeesha

I tried a full shutdown. I let it go full "red" before clicking the start icon. XO is still reporting very different MAC and IP assignments from what the guest OS has configured. Is it perhaps due to the tools that OPNsense packages?

I have my router (OPNsense) running in a VM on XCP 8.2 and there seems to be a disparity between what XO is reporting from the guest utils and what the guest OS itself is saying.

In the guest OS, interface with MAC ending in 72:1e is my WAN interface, getting a DHCP address from my ISP. In XO, that interface is a static IP.

Here are two example interface assignments in the guest OS:

Here is how each of those is configured:

Here is how XO is reporting them:

That interface with the Jumbo Frames configured should be a 10.10.10.X address, per the OS config. That 192.168.5.X is on a completely different interface in the OS, and the MAC X:X:X:X:72:1e should be having the address currently shown on X:X:X:X:fd:93

Any suggestions? Because resolving networking issues when I can't get the information sources to align is problematic.

Thanks!

I have a couple of networks created for VLAN tagging purposes. These networks show on the Pool networking tab. They also appear on the two hosts that aren't the master. They are missing from the master node's network tab. Now, I can't migrate any VMs that have interfaces on these networks to the master host.

Is there a way to recreate those networks on just the one host? Being the master that is missing them, I'm afraid they'll disappear altogether at some point and those VMs will have issues.

Thanks!

@olivierlambert Fair enough. I will bring the disk for this VM onto the local SR for patching and rebooting my file server that hosts the NFS SR, just to be on the safe side.

I was hoping I could minimize the downtime by live-migrating the disk onto the host running it, but get the same error about PCI hardware as if I were trying to migrate the whole VM to another host.

But as far as you know, there is no technical reason for why it wouldn't work, like there is for live-migrating between hosts with passed-through hardware.

I have a VM that has a GPU assigned via PCIe passthrough, with its disk on an NFS SR. I understand why I can't live migrate a VM that utilizes PCIe passthrough.

But what is it that prevents migrating just the virtual disk to the local SR on the host running that VM?

@Darkbeldin I wasn't sure about turning on two-factor. I went through that process and generated the "app password" and it seems to be working.

Thanks for the info!

With Google essentially shutting off "less secure app access" in May, is there anything on the horizon for the email plugin to still function with Google's SMTP? I noticed my alerts stopped working, and when I checked my Google dashboard for the account that sends the alerts, Google had shut off the "less secure 3rd party apps access" automatically, and turning it back on was accompanied by a warning that it was being turned off permanently on May 20, 2022.

@Danp No, but I seem to have figured it out. Its a weird one, considering the platform.

These are HP DL360s, with fully licensed iLOs. But ... the CPU errors are gone if I physically connect a keyboard to the server. If I reboot just monitoring via remote console in the iLO, I get the errors. If I go to the machines and connect a USB keyboard before the reboot, then go back to my workstation and do it all through XO and watch via remote console, they come up fine. My post earlier about two coming up fine, I remembered that coincidentally, I'd switched the KVM to them.

So just an FYI to anyone who might have the same problem, plug in a keyboard.

@stormi Yeah, I spoke too soon. Two of my hosts came right up fine after the patches. The third has been in a boot loop with CPU panics and fatal page faults.