I just noticed that in the couple weeks since this happened, a new set of Pool Patches was available, so I ran through the usual process to update those and had no issues at all. This time, everything went smoothly, just as it has for the entire time I've had this lab up and running.

So, chalk this up to one of those weird glitches you're bound to see when you use any technology long enough.

As far as I'm concerned, the issue has been resolved. Please feel free to close out the issue.

• https://github.com/Jarli01/xenorchestra_installer
• https://github.com/ronivay/XenOrchestraInstallerUpdater (forked from above. I use this one in my homelab, personally)

These tools are given no official support from Vates as they are 3rd party tools, and as per Vates, you ought not be using the compiled from sources version in production, so YMMV..

IIRC Jarli01 is an active user on this forum but they have a different username here that I can't recall.

If nothing else, scripts like these make the process consistent which makes troubleshooting MUCH easier when things do go wrong.

Just a bit of food for thought.
Cheers

I cannot say what the issue was. However, when I got everything up and running today, just for giggles I started the Orchestra server back up and ran through the usual update process to see if maybe, just maybe, one of the Commits between when I took the servers down three weeks ago and today would fix the issue. Lo and behold, it did. After applying the latest Commits (currently fb0e1) I was finally able to get back into Orchestra normally. Everything appears to be running exactly as it should be now.

I have no explanation; other than maybe something went sideways when I last applied Commits that somehow went "sideways" and was corrected by this latest application.

The really weird thing is nothing has changed when I check services, like @knightjoel mentioned. Even though EVERYTHING is now working properly in Orchestra, I still see the same messages when I check the Orchestra service status:

So... I don't know what is happening, exactly... other than everything is working again beautifully!

@JamfoFL said in Xen Orchestra from Sources unreachable after applying XCPng Patch updates:

This is very odd. When I check to see if the Orchestra status is running, everything looks OK:

This doesn't look ok. The process behind this service has actually exited. Suggest you start/restart this service and check if XO is reachable. If it's not, dive into the logs and look for clues why the process is exiting. You may want to also try manually running /etc/init.d/orchestra and see if that produces any helpful output.

But when I try to run the command you sent over, I get an error message stating "Unit xo-server.service" could not be found". However, when I check in the very same folder from which I am running the command, I can see xo-server.service right there.

Not taking away from the points others have made about .service files needing to be in the correct location, but in your case, worrying about the xo-server.service file is probably a dead end. It appears whoever installed XO created the systemd service as orchestra. You needn't try and "fix" the fact you don't have an xo-server.service.

.joel

@john.c I am able to PING the VM hosting Orchestra with no issues.

How about trace routing the tcp port using tcptraceroute (Linux) or tracetcp (Windows). Can you run this on the IP Address and port used by Xen Orchestra, from both local address (same machine) and another address (remote machine)?

Can help to see if there’s an issue in the path to that IP’s TCP Port!

@john.c Yeah, I'll probably have to suck it up and build a new one. However, it should be noted that I've in no way done any kind of odd customizations to Orchestra. When I installed it way back when I followed the explicit instructions right from the https://docs.xen-orchestra.com/installation#from-the-sources site and that was it. I think the instructions were slightly different back then, but I've never done anything other than what is in the instructions... I'm not "sophisticated" enough with Linux flavors to do any kind of tinkering on my own.

It may be a while till I get a chance to do a complete rebuild... but I'll update here once I've completed!

How about to the XCP-ng hosts or their network connections?

It may be a while till I get a chance to do a complete rebuild... but I'll update here once I've completed!

Hi,

No, there's no reason XCP-ng patches are doing issues like this (at least I'm not aware with our customers and our own install is also patched too: no issues).

This makes me believe it's a configuration/environment issue on your end. I know it doesn't help, but at least it's XCP-ng unrelated

If he didn’t have this issue before, but does now and it worked beforehand. While you and other customers don’t have the issue as well as possibly other XO from Sources users.

Could it be something in one of those patches, that’s interacting with his configuration and environment used by his infrastructure and/or XO From Sources VM, causing his issue?

Knowing what it it’s in his environment could be helpful as it can be documented for future reference by him and/or Vates. Both to warn people with similar configurations and also incase other surprises crop up in unexpected places!

@john.c Yes, the xo-server.service is on the exact, same VM where Xen Orchestra is running. It's actually in the same folder: /xen-orchestra/packages/xo-server.

When I run systemctl for the sshd.service as you recommended, I get the following:

Here's what's really, really weird, however. I have a second demo system: completely separate hardware, installations, the works... this second demo system is almost identical to the demo system on which I am having the issue and this system is working just fine. When I go to the host VM for Orchestra on that server it has the same message that it can't find the xo-server.service! And yet, just to reiterate, that version of Orchestra is working just fine.

I'm almost wondering if the updated XCPng patches broke something, as the only difference between the two setups is I haven't updated the XCPng patches on the second set of hardware, yet.

Could you try relocating it to the sshd.service path location?

Though if it requires alterations to the xo-server.service to make it find the necessary executables, make them.

Don’t forget “systemctl daemon-reload” to reload service files.

Anyway that Orchestra.service file (Orchestra) file seems a bit odd! Did you create or use this to launch Xen Orchestra, in the past as a daemon automatically?

This maybe why Xen Orchestra is running without its working xo-server.service file!

If that still causes issues it maybe worth rolling a new instance of XO From Sources, but in an environment closer to the one used by Vates to build XOA appliance.

No, there's no reason XCP-ng patches are doing issues like this (at least I'm not aware with our customers and our own install is also patched too: no issues).

This makes me believe it's a configuration/environment issue on your end. I know it doesn't help, but at least it's XCP-ng unrelated

When I run systemctl for the sshd.service as you recommended, I get the following:

Here's what's really, really weird, however. I have a second demo system: completely separate hardware, installations, the works... this second demo system is almost identical to the demo system on which I am having the issue and this system is working just fine. When I go to the host VM for Orchestra on that server it has the same message that it can't find the xo-server.service! And yet, just to reiterate, that version of Orchestra is working just fine.

I'm almost wondering if the updated XCPng patches broke something, as the only difference between the two setups is I haven't updated the XCPng patches on the second set of hardware, yet.

@john.c How do I locate that specific path? And, this instance of Orchestra has been running flawlessly for almost two years now and nothing was changed other than applying the patches for XCPng.

Where do I look to make sure that the proper path is specified?

The xo-server.service file needs to be on the VM if that’s where you have Xen Orchestra. You can find out the appropriate location by querying where you have other working daemons by using systemctl or running a find for the chosen .service unit file. It needs to be other than the Orchestra.service and xo-server.service.

A good one to try if it’s enabled and running is sshd.service!

Are you sure that there hasn’t been anything done to the XO from the sources? Also have you tried rolling a fresh install of XO From the Sources in case something went fubar (Gang Aft Agley)?

Where do I look to make sure that the proper path is specified?

@Danp Thanks for getting back to me.

This is very odd. When I check to see if the Orchestra status is running, everything looks OK:

But when I try to run the command you sent over, I get an error message stating "Unit xo-server.service" could not be found". However, when I check in the very same folder from which I am running the command, I can see xo-server.service right there.

Not sure why it would do that. Any suggestions?

Thanks!

Systemd daemons (services) need to be in a particular location and loaded into systemd in order to work. If it’s outside of the specific path for the OS then it won’t be found by systemd and systemctl.

This is very odd. When I check to see if the Orchestra status is running, everything looks OK:

But when I try to run the command you sent over, I get an error message stating "Unit xo-server.service" could not be found". However, when I check in the very same folder from which I am running the command, I can see xo-server.service right there.

Not sure why it would do that. Any suggestions?

Thanks!

It was installed following the specific directions on the https://docs.xen-orchestra.com/installation#from-the-sources site.

You can also check the recent logs with journalctl -u xo-server -f -n 50