@CodeMercenary Glad it worked!

I just noticed that in the couple weeks since this happened, a new set of Pool Patches was available, so I ran through the usual process to update those and had no issues at all. This time, everything went smoothly, just as it has for the entire time I've had this lab up and running.

So, chalk this up to one of those weird glitches you're bound to see when you use any technology long enough.

As far as I'm concerned, the issue has been resolved. Please feel free to close out the issue.

@CodeMercenary Sorry I have not replied to you sooner. Unfortunately, the timing of this issue was bad! I was in the process of moving and had to tear down my demo equipment for the move and have just now reassembled everything and got back up-and-running.

I cannot say what the issue was. However, when I got everything up and running today, just for giggles I started the Orchestra server back up and ran through the usual update process to see if maybe, just maybe, one of the Commits between when I took the servers down three weeks ago and today would fix the issue. Lo and behold, it did. After applying the latest Commits (currently fb0e1) I was finally able to get back into Orchestra normally. Everything appears to be running exactly as it should be now.

I have no explanation; other than maybe something went sideways when I last applied Commits that somehow went "sideways" and was corrected by this latest application.

The really weird thing is nothing has changed when I check services, like @knightjoel mentioned. Even though EVERYTHING is now working properly in Orchestra, I still see the same messages when I check the Orchestra service status:

So... I don't know what is happening, exactly... other than everything is working again beautifully!

@john.c I am able to PING the VM hosting Orchestra with no issues.

@john.c Yeah, I'll probably have to suck it up and build a new one. However, it should be noted that I've in no way done any kind of odd customizations to Orchestra. When I installed it way back when I followed the explicit instructions right from the https://docs.xen-orchestra.com/installation#from-the-sources site and that was it. I think the instructions were slightly different back then, but I've never done anything other than what is in the instructions... I'm not "sophisticated" enough with Linux flavors to do any kind of tinkering on my own.

It may be a while till I get a chance to do a complete rebuild... but I'll update here once I've completed!

@john.c Yes, the xo-server.service is on the exact, same VM where Xen Orchestra is running. It's actually in the same folder: /xen-orchestra/packages/xo-server.

When I run systemctl for the sshd.service as you recommended, I get the following:

Here's what's really, really weird, however. I have a second demo system: completely separate hardware, installations, the works... this second demo system is almost identical to the demo system on which I am having the issue and this system is working just fine. When I go to the host VM for Orchestra on that server it has the same message that it can't find the xo-server.service! And yet, just to reiterate, that version of Orchestra is working just fine.

I'm almost wondering if the updated XCPng patches broke something, as the only difference between the two setups is I haven't updated the XCPng patches on the second set of hardware, yet.

@john.c How do I locate that specific path? And, this instance of Orchestra has been running flawlessly for almost two years now and nothing was changed other than applying the patches for XCPng.

Where do I look to make sure that the proper path is specified?

@Danp Thanks for getting back to me.

This is very odd. When I check to see if the Orchestra status is running, everything looks OK:

But when I try to run the command you sent over, I get an error message stating "Unit xo-server.service" could not be found". However, when I check in the very same folder from which I am running the command, I can see xo-server.service right there.

Not sure why it would do that. Any suggestions?

Thanks!

@Perplexed9828 It's installed on one of my two hosts, but I've tried moving it to the other host with no love.

It was installed following the specific directions on the https://docs.xen-orchestra.com/installation#from-the-sources site.

Good afternoon!

It is currently July 8, 2025. I received the usual notification this morning that new patch updates were available for XCPng. I am currently running XCPng version 8.3.

Prior to updating the XCPng patches, I also made sure to update Xen Orchestra to the latest available commit; this would be the latest available commit as of 12:00 PM EST on July 8th. I would love to tell you the specific number, but the issue is after applying the latest XCPng patches and rebooting the physical hosts, I can no longer connect to Orchestra.

The Linux host for Xen Orchestra boots just fine, I can log on and everything seems OK. However, I don't think Xen Orchestra is running, as none of the automated tasks appear to be working, either.

Is there something in the new XCPng patches that prevents Xen from working? Do I need to make some changes to the Xen Orchestra configurations?

Thanks!

@Andrew Interesting... up until this time we've never needed to completely rebuild XO every time a new Commit was released or a Node update was required. The existing installation just kept working. I hope this isn't something that is going to become standard as having to completely rebuild XO every time a new Commit is available would be a bit of a pain...

@Andrew I guess maybe that's the difference? I update Node and then do the usual update process (git checkout, yarn, yarn build, etc...) and my backups were still unable to run without Node 18 installed. Sounds like you went completely back to square one and built XO as if it were a brand-new machine.

@Andrew Oh... you rebuilt XO from scratch?

@Andrew Do you still have Node 18 installed even though you are on Node 22 now? When I upgraded to Node 20 I uninstalled Node 18 and immediately all backups started to fail. I received error messages about needing Node 18.19.1 for the backups to work.

I reinstalled 18.19.1 and that got everything working again. So, it seems that, for whatever reason, that newer versions of Node are supported, but you have to keep the older versions around or some things will break. In fact, I found if I installed the latest version of Node 18, 18.20.1, that my backups still failed. I specifically had to use Node 18.19.1 to keep things working.

I just wanted to post an update as of Monday, February 24, 2025...

I updated to Commit f18b0 and that seems to have fixed the issue. All of the dropdown options that were missing in the above screen shots, as well as many others, are all back and responding normally once again. So, it looks like the issue has been addressed and operations have returned to normal.

The only oddity I now see after all of that is when I run yarn after pulling the updates, I receive an error message that the expected version of Node is now v20. I upgraded to v20, but it seems that the backup portion of Xen still needs v18... so be sure to keep both on the system or things will break. I have both v18 and v20 installed and running, with v20 set as default.

Other than that little oddity, it seems everything else is doing everything it should be doing!

Thanks to the excellent Vates teams for getting everything back on track.

@jr-m4 I just wanted to report that today, February 24, 2025, that I updated to Commit f18b0 and that seems to have fixed the issue. I am able to see all drop downs once again and everything seems to be working normally.

@Danp and Support Team... thank you for looking into the issue!

@probain Thanks for the update... I will monitor that ticket, as well.

XCP-ng 8.3

Xen Orchestra from Sources
Commit 97230

After updating to the latest commit shown above, all of the drop-down boxes are empty on all screens.

Everything else seems to be working just fine... it's only an inability to go into any drop-down on any screen to see the available options, machines, backups, etc.

Any ideas on what to look for?

Thank you!

@MathieuRA Thanks so much! I appreciate all the effort!

@MathieuRA Yes, I can confirm I am using the All Running VMs and All Running Hosts (I am not using All Running SRs, but I never get alerts for those because I have a LOT of free disk space).

I did place an exclusion for one of my VMs (the one that was generating dozens and dozens of alerts) to cut down on some of that chatter, but even with one machine excluded, when I do get a report from one of the other VMs it still has the same issue: the proper VM will generate the alert, but an improper VM will be reported in the end of alert message.

So... as far as I can tell, we still have the issue with the improper machine identification and the Average Length field is ignored so a machine that pops over the threshold, then briefly under the threshold for a few seconds, then back over the threshold again will generate three messages (alert, end of alert, alert) in several seconds instead of looking at the average to make sure the dip isn't just a brief one.

Hopefully that makes sense.

Thanks again!

@frank-s Think of it like this...

You take the original backup with snapshot on October 1st.

You then have a differential backup on October 8th. That differential only writes the changes that make the state of the server on October 8th different (hence the term differential) from the state of the server as it was on the full snapshot on October 1st.

Now you delete the snapshot of the full system state taken on October 1st, leaving only the October 8th differential.

It's now October 15th and you want to restore something from the October 8th differential and you figure you'll just make a new full snapshot from October 15th and then try to apply the October 8th differential. However, this won't work because the differential from October 8th will be missing key reference points it will be looking for in order to overlay its changes onto the snapshot. What else on that server changed from October 1st to October 15th that isn't contained in the differential? What changes occurred from October 8th to October 15th that are newer than what is on the differential? You could, potentially, brick your entire system if the differential started overwriting changes it sees from its system state that are actually newer than the data it contains.

Differentials are entirely dependent on the full snapshot on which they are based. Any new snapshot taken after their creation will be totally foreign to the differential... those differentials will be looking for very specific system states that existed at the time the original snapshot was created, and those will be completely different from the system state as it will appear on a new snapshot created after the differential.