Hard reboot sorted itself out

@Danp

Thanks for that command, I had a similar command I found but it didn't display the timestamp fully so I couldn't pinpoint the time of the message, but I had a feeling this was there the issue was.

[Fri Aug  7 16:26:44 2026] CIFS VFS: Server 192.168.x.x has not responded in 120 seconds. Reconnecting...

Lots of these types of messages since yesterday.

And df -h does hang as that's something I was testing yesterday when debugging to see if some disk had got full that I hadn't spotted.

There has definitely been a few network disconnects over the past 24 hours as I was testing various things which requires a bit of fiddling with the cables and jumping between networks, and it's quite possible that this happened during the backup window but I can't recall for sure.

The Storage Repositories are all reporting as working correctly in XO, XOA, XCP-ng Centre.

What's the next steps to fix this?

I assume there is a way without deleting the Storage Repository in either XO/XOA/XCP-ng Centre and re-attaching it?

Same behaviour across XO, XOA + XCP-ng Centre.

Not had this before, I'm struggling to restart VMs. I first noticed that some of the backup tasks yesterday were running way longer than they normally do, and then still running into today which is when I started investigating the issue.

The biggest clue at the moment is when running the command;

xe task-list

uuid ( RO)                : 92b59f22-15ae-2551-8839-f0f938a07af7
          name-label ( RO): SR.scan
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : 880f0f4a-8e84-0954-2521-81831d559dd3
          name-label ( RO): Async.VM.hard_shutdown
    name-description ( RO):
              status ( RO): pending
            progress ( RO): 0.273

Tried;

xe-toolstack-restart

Made no difference.

Just seeing if there is any more debugging I can do to try and get this working without doing a full host power off/on.

It's all strange though as the running VMs are all working fine.

The Storage Repository it appears to be getting stuck on is the one storing the ISOs for the VMs. Yet nothing has changed at all.

The Host can ping the Storage Repository fine too.

Any ideas what to test to get this back working without a full Host reboot?

I've been round the loop many times of killing the stuck tasks and then trying to Start/Stop VMs and same issue each time, just goes back to the same hung state.

Storage Repository is on a Windows machine, so the only thing that is jumping to mind is some janky automatic Windows update that has kicked in and broken something. But it's strange the running VMs are still working with their ISOs sitting in that same Storage Repository (I'm not 100% clear off the top of my head if running VMs still require access to the ISO Storage Repository to be running post-boot)

Commenting for my future self and others as I just came across this issue too and it's not overly well documented and comments on the forums are often missing a a full story for how to debug and get this working......

When using Ubuntu 24.04, the usual apt method of installing XE Guest Utilities via the command;

apt-install xe-guest-utilities

Works, but doesn't. It successfully brings through the Memory information through to XO/XOA/XCP-ng Server, but not the IP Address, which is a tad strange.

When checking the version of xe-guest-utilities that is contained within the APT Repository that comes configured out of the box with Ubuntu 24.04 using the following command;

apt info xe-guest-utilities

You can see the response is that it is running version "7.20.2-0" (as at the time of writing, which is before running an apt update command.

Even when after running an update command;

sudo apt update

Then checking what packages are available to be updated using the following command;

apt list --upgradable

xe-guest-utilities wasn't available for an update past this point unfortunately.

So to uninstall xe-guest-utilities installed via APT, simply run the following command;

sudo apt remove xe-guest-utilities

Then install via XO/XOA/XCP-ng Guest-Tools.iso Mount + Command Line by following these steps. Firstly, Select the guest-utilities.iso from the Disk within the XO/XOA/XCP-ng Platform for the VM. Then SSH into the box/console view and run the following commands;

sudo mount /dev/cdrom /mnt
sudo bash /mnt/Linux/install.sh
sudo umount /dev/cdrom

What you will notice when you run the second command is that gets installed here is "xe-guest-utilities_7.30.0.12_amd64.deb" - So is further ahead.

Then you'll need to use the following commands to start the service on boot and also start the service.

systemctl enable xe-linux-distribution.service
systemctl start xe-linux-distribution.service

And that now results in the IP Address of the virtual machine flowing through to XO/XOA/XCP-ng, which ever you are using.

@Greg_E Interesting, I've not had a play with RPUs yet, sounds pretty handy though.

I can see how that could work when using Truenas as the HostOS on the Bare Metal, whereas I've got XCP-ng on the Bare Metal so I'd need to get some more machines to give that a go in due course.

@Greg_E Looks like you were writing a reply as I was...

But yeah, Truenas is on the roadmap at some point in the future, once I've had the time to dig into the details and understand where this fits in the overall system architecture.

Out of interest though, where would you see Truenas fitting in with this setup outlined below?

My initial thoughts are that Truenas in this context would be either;

1. A glorified USB Stick storing ISOs
2. A glorified (RAID Redundant) of an External HDD running as the Backup platform
3. Or both of the above

Have I understood that context correctly?

Current setup is essentially.....

• Bare Metal
• • HBA Raid Controllers for Storage
• • • Primary Raid Array
• • • Backup Raid Array
• • Dom0
• • • XCP-ng Server
• • DomU
• • • XOA VM
• • • • Backups; Primary Raid Array -> Backup Raid Array
• • • • Backups; Primary Raid Array -> Remote PC
• • • • Backups; Primary Raid Array -> Dom0
• Remote PC
• • ISOs via SMB Share

Truenas in the above context would simply be an additional backup to either;

1. A Truenas VM on the Base Metal as a VM
2. A physically separate Bare Metal running Truenas (aka. a glorified RAID'ed External USB Disk).

Is that where you would see things fitting into the overall system architecture?

Curious to get your thoughts on that.....

Solved.....

Turns out that everything above was basically a red herring.....

After a lot of playing around with the carrot and stick, noticed a lot of Tasks in XOA just getting stuck at 0%, the kinds of Tasks that usually just appear and disappear in no-time, to the point that you just don't even notice they are happening really. Something just felt off.

Multiple restarts of the following command when SSH'ing into the XCP-ng Server

xe-toolstack-restart

It cleared the Task list in XOA, but things just started to back up again with no obvious insights into what was really going on. I did notice the Progress % Bar wasn't showing up though, that was odd....

Anyhow, after all this "fun" I decided to give the server a physical kick, aka. a Manual Hard Power-Off and Power-On.

Low and behold, the age-old IT solution of turning it off and back on again has solved everything magically #FFS

Hopefully the above self-documenting debugging process helps future readers (myself included, more than likely)

Anyhow, just a thought @olivierlambert @greg_e - I've clearly no idea what fell over under the hood here, you guys probably have much more of an insight into this than I do - So the question is.....

1. What went on here? (gut feel)

2. What insights could be surfaced to the user of XOA/XO to help debug these things?

3. What friendly error messages could be added to help with #2 etc.

Kind of rhetorical questions really, so just adding some thoughts for future product development.

Personally, my gut feel here is that it's probably some low level Linux service/package/driver/etc. that fell over and didn't auto-recover - and XOA/XO had no way of detecting what had happened and hence was just a bit blind so seemed to go a bit wacky.

If it helps to improve the platform, happy to have a private DM or in this thread re. any kinds of commands I can run on said server to see if it can help pinpoint what failed (if logs persisted after a hard reboot) - it'll mean more to you than me the outputs of said commands.

SMB v2 + 3 are also turned on;

Get-SmbServerConfiguration | Select EnableSMB2Protocol

Which outputs

EnableSMB2Protocol
------------------
              True

Bit more info.....

The issue reading up on this appears to be related to SMB v1, which is commonly referred to as "Insecure" (More like Less Secure if we're being accurate.....)

I've just run the following on the Windows machine where the SMB Shares live via Powershell (as an Admin)

Get-WindowsOptionalFeature -Online -FeatureName SMB1Protocol

Which has clarified that this is currently running, which throws a spanner in the works as I was expecting that to be Disabled, which would back up the theory in the previous comment.

FeatureName      : SMB1Protocol
DisplayName      : SMB 1.0/CIFS File Sharing Support
Description      : Support for the SMB 1.0/CIFS file sharing protocol and the Computer Browser protocol.
RestartRequired  : Possible
State            : Enabled
CustomProperties :
                   ServerComponent\Description : Support for the SMB 1.0/CIFS file sharing protocol and the Computer
                   Browser protocol.
                   ServerComponent\DisplayName : SMB 1.0/CIFS File Sharing Support
                   ServerComponent\Id : 487
                   ServerComponent\Type : Feature
                   ServerComponent\UniqueName : FS-SMB1
                   ServerComponent\Deploys\Update\Name : SMB1Protocol

Various similar issues I've spotted on Reddit re. Windows updates breaking SMB Mounts over the last 12-18 months with updates. Impossible to track down the specifics so not even going to bother.

Then found this issue too, https://xcp-ng.org/forum/topic/10545/long-delays-at-46-when-creating-or-starting-a-new-vm

Which was solved by moving from SMB Mount to NFS Mount.

That isn't going to work with my infrastructure setup as the Windows machine is on a Home Edition, and NFS is only supported on Business or Enterprise edition.

As it stands right now, and given nothing I've tried to get the SMB Mount to work again, feels like I'm at a dead end to get this fixed. Looking like I'll need to re-address the architecture for storing the ISOs and one of the backup routes (thankfully there are many types of backups in place for redundancy)

Not the end of the world, but a tad annoying which feels like it's probably a Windows update that broke this with one of the almost daily automated updates/shutdowns/restarts that kicks in. Only started happening yesterday, all previous backups via the SR were working fine until then. That's the only thing I can put this down to really.

Northing weird getting blocked on XCP-ng firewall for outbound connections

iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_input_chain  tcp  --  anywhere             anywhere             tcp dpt:nbd
ACCEPT     gre  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:rtps-dd-mt
RH-Firewall-1-INPUT  all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:rtps-dd-mt

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_output_chain  tcp  --  anywhere             anywhere             tcp spt:nbd

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere             icmp any
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW udp dpt:ha-cluster
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:21064
ACCEPT     udp  --  anywhere             anywhere             multiport dports hpoms-dps-lstn,netsupport
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain xapi_nbd_input_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain xapi_nbd_output_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

All the above is just out of the box default, no additional configurations.

Hmm.....

yum install -y nc

nc -zv {IP of Windows Machine where SRs live} 445
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to {IP of Windows Machine where SRs live}:445.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

Is that suggesting with 0 bytes sent that it's not getting out from XCP-ng? So never actually hitting the Windows machine where the SMB SR lives?

What could be causing that?

@Greg_E Looks like I'm still on XOA 5 as I can see the "Try XOA 6" in the side menu.

Just tried to create a VM via XCP-ng Centre, getting the same issues.

Can't create or Delete VMs.

I've SSH'd into the HostOS, and I can successfully ping the Storage Repository.

Then just tried removing the existing Storage Repository where the ISOs are hosted, and it wouldn't disconnect, just hanging again.

Tried to create a new Storage Repository (same as the old ones for the ISOs) and that wouldn't connect.

Certainly feels like something SR related.

Just noticed too that one of the backup jobs that sends the backups to a different SR on the same machine where the ISOs live also failed.

Something has happened yesterday which is making the HostOS now no longer able to connect to the SR.

I've just tried disabling Norton Firewall on the machine where the SR live, as that has got in the way historically, but that made no difference.

Also just checked the Windows CIFS / SMB Settings and noticed that one of the three options was unchecked (previously all three where checked and I know that has not been disabled manually) - So will give a reboot a go to see if that solve things.

Bit baffled what else to try to get the SR re-connected as literally nothing manually has changed, so it has to be some Windows or Norton janky auto update that has kicked in and screwed things up.

Hmmm.....

Bit baffled by this one.....

I've spun up a lot of VMs historically with no problems at all.

Yet when I've tried recently on the XOA from Sources (installed on a VM on the Bare Metal Host)

It's failed on 3x attempted VM Creations, and their subsequent Deletions have also failed to kick in.

Initially though this could be a network issue as backups were taking place at the same time, but even after those finished (well, 99% finished according to the Tasks) - All of these basic tasks were just getting stuck on the process.

VM.start: 54% (Stuck at....)

Async.VM.hard_shutdown (on HOST) 33% (Stuck at....)

VBD.unplug (on HOST) 0% (Stuck at.....)

Async.VM.destroy (on HOST) 0% (Stuck at.....)

As I mentioned, a handful of backups are stuck at 99%, so not sure if that is somehow related and/of blocking things - But can't see how it would be as the backups are only for currently running VMs)

It's all a bit odd.....

And while odd is normal working in tech, I'm getting zero information from XOA about what the issue is, and ultimately how to solve it, and a quick Google of "Turn the Host off and back on again isn't really an option in the real world"

Any ideas?

Regards,
Michael

That would explain things, and also why my latest test using AlmaLinux 10 also failed in the exact same way that Rocky Linux 10 did.

I just wish I read this reply 12 hrs ago

Think I'll give this stuff a go on v9 of Rocky Linux and AlmaLinux to check things are working, which I imagine will work fine.

I've just confirmed the EPEL version running on AlmaLinux 10 at the time of writing this comment is indeed v 10.1

via the command;

sudo yum repoinfo epel

@dknight-bg What is the best way to keep track of this stuff proactively, i.e. What would I need to look at so that I could have determined what you have commented myself?

Commenting for my future self.

This works for CentOS Stream 10;

# Install Extra Packages for Linux, where xe-guest-utilities-latest lives
sudo yum install epel-release

# Install
sudo yum install xe-guest-utilities-latest

# Start the Service on Boot
sudo systemctl enable xe-linux-distribution

# Start the Service Now
sudo systemctl start xe-linux-distribution

# Check the Service is Running
sudo systemctl status xe-linux-distribution

# Check it's installed
sudo yum list installed xe-guest-utilities-latest

I wasn't particularly committed to Rocky Linux 10, so thought I'd give the usual steps a run on a different similar OS using CentOS Stream 10. Works absolutely fine.

As a comparison, when I run the above steps on Rocky Linux 10, it fails on the second step above when trying to install from yum, which appears to be due to this not being included in EPEL.

sudo yum search xe-guest-utilities

No matches found.

Curious to understand if anyone knows why EPEL is different between CentOS Stream 10 VS Rocky Linux 10. Is this just some upstream thing that isn't perfectly aligned?

Struggling to find a full and complete example of how to do this, and everything I've tried that has worked before and via docs I've found just isn't working.

I'm probably missing something fairly obvious for those who have done it before.

Tried via XCP-ng Centre (8.2, with all patches applied) - Seems to be some weird bug in there from what I've seen on the forums with the guest-tools.iso mount not working.

Tried via XOA with guest-tools.iso mount, and then via console to run commands - Which didn't work

Tried via XO from Sources with guest-tools.iso mount, and then via console to run commands - Which didn't work.

Followed guidelines from here https://docs.xcp-ng.org/vms/#linux-guest-tools

But it's not complete documentation, I think the missing bit of info is "enable the appropriate repository" which if documented, then it should mean that things can be installed via Yum with

yum install xe-guest-utilities-latest

But unsure the steps to take to configure something this ambiguous.

I'm also a bit confused why the guest-tools.iso isn't working like before. I can only assume that it's because Rocky Linux 10 is newer than XCP-ng Server 8.2, so it's probably an alignment issue.

What am I missing to get this working?

Also curious what the use case of this is.

The only genuine use case I can think of for this is for a basic Storage Repository containing ISOs for VMs so they can spin up.

Or are you expecting an AWS S3 equivalent or Windows File Explorer equivalent?

Thanks @lawrencesystems - I've finally got around to watching that video, explains the content well, quite interesting to see that happening in real time when the RAM is being land-grabbed by other VMs when things are getting full based on the different configuration settings.

@tjkreidl That sounds like it's answered 99% of initial questions to be fair.

The 1% I'm not clear on is, what would happen in the scenario whereby say I have 10x VMs, which in total took up 100% of RAM for example - but they were all turned off.

Would I still be able to create an 11th VM with 100%+X RAM allocation and still turn it on successfully? i.e. because the other 10VMs are turned off?

From what you've said, I'd come to the conclusion of 'yes'.

Seems easier to have a conversation around the topic rather than me trial and error playing around blindly.