@olivierlambert said in Commands in Xen Orchestra Jobs no longer working:

The best solution is to rely on XOA, you know

I agree with @olivierlambert! Personally, I'm using XO from sources in my home lab environment -- nothing is "production" and I'm mostly having fun trying to give back to the open source community.

@olivierlambert said in Commands in Xen Orchestra Jobs no longer working:

The best solution is to rely on XOA, you know

I agree with @olivierlambert! Personally, I'm using XO from sources in my home lab environment -- nothing is "production" and I'm mostly having fun trying to give back to the open source community.

I have seen this as well, and I reported Basic Job failing with "Invalid parameters" Error on vm.start #5983.

As of this writing, that bug appears to be in Triaging status.

I would also like to note that I have a startup job that was configured some time ago (didn't document when) which works just fine. If I recreate the same job from scratch, I get this new error.

This suggests that the issue is related to the newly created job object and the previous job object is still sending a string value containing the one VM rather than an array with a single vm.

It seems that either the UI changed to allow selection of multiple VMs per job, or if that was already possible, then the UI is attempting to send an array rather building a comma-separated string for example.

Hopefull you have a backup/snapshot to which you can revert until the issue is resolved if you really need that functionality.

@olivierlambert -- this may be related to xcp-ng bug 5929, but perhaps I don't know what types of traffic I need to allow through my firewall xo allow xoc to communicate with my storage network.

I am again on the latest commit to master as of today.

My Default Migration Network was set to my Storage network (10.0.60.0/24).
My storage server is 10.0.60.2 which hosts my default NFS SR as well as my xo nfs remotes.
xcp-ng-1 management interface is 10.0.10.11 with 10.0.60.11 on storage net
xcp-ng-2 (pool master) management interface is 10.0.10.12 with 10.0.60.12 on storage net

My xoc instance only had one interface on management network with address 10.0.10.10 and backups didn't work.

I added firewall rules allowing 10.0.10.10 to access 10.0.60.11 and 10.0.60.12 via tcp/udp 111,443, and 2049, but that made no impact on backups -- I saw via pftop that 10.0.10.10 (xoc) was contacting my pool master, xcp-ng-2 via 10.0.60.12:443 when I tried to start the backup. I didn't pay super-close attention, the connections did not stay active and disappeared after 30 seconds - 1 minute

I then added an interface to xoc on my storage network with address 10.0.60.10 to avoid the firewall altogether and started a CR job which proceeded almost immediately.

I then disconnected the xoc interface on the storage network, configured my default migration network back to the Management network, disabled my firewall rules and the backups worked again.

As per @julien-f in xcp-ng bug 5929 xo uses the pool's default migration network for stats, backups, etc. as of 5.62 and that it might not have been a good idea because xo might not always be able to access the storage network. Wouldn't xo always want to communicate with the pool master via the master's management interface, even if management interface is not on the default migration network? I had always assumed that hosts only listen for xapi commands on the management interface's IP -- is that correct or a misunderstanding on my part?

@olivierlambert Today I created a new key and certificate as per the official documentation and I added SANs for the hostname and IP address of each xcp-ng hosts in my pool. I used XO's Replace existing certificate button, I saw the correct fingerprint apear on both hosts, I restarted the toolstack on both hosts, and I verified via cli on one of the hosts that the fingerprint of /ext/xensource/xap-ssl.pem matches that which is displayed in xo.

Nonetheless, I get the same results in the two instances of xo: the clone of the snapshot I took before updating (on commit feat: release 5.63.0 (#5925)) connects just fine with Unauthorized Cerificates disabled, and the original instance that was updated will only connect when Unauthorized Certificates is enabled, otherwise it gives the error self signed certificate in certificate chain.

I then ran the update tool again and it did indeed move to what is as of this writing the latest commit at https://github.com/vatesfr/xen-orchestra: [info] Updating xen-orchestra from 'abcabb736' to '9ceba1d6e' however, I still get the same behavior.

I'm not sure what, but something happened after 5.63.0 that is causing xo to reject the certificate even though I'm using the same certificate chain as before.

@olivierlambert thank you for that information!

I have been creating a unique certificate for each host, and that explains why the requests have been failing / hanging. Based on what you're saying, I ought be making one certificate that covers all hosts in a pool and applying that same certificate and key to each host, yes?

For example, a certificate with CN="Example Domain XCP-NG Hosts" and SANs xcp-ng-1.example.com, 10.0.10.11, xcp-ng-2.example.com, 10.0.10.12, etc.

I'll give that a try and follow up.

I have a pool with 2 xcp-ng 8.2 hosts and on the same management network I have a debian 10 based VM within which I have built xo from sources using https://github.com/ronivay/XenOrchestraInstallerUpdater.

I had previously followed the guide TLS Certificate on XCP-NG as well as the guide for additional configuration to specify on XenOrchestra side, but I was getting Xapi#getResource /rrd_updates tasks lingering if I connected to my host when Unauthorized Certificates was disabled but not when connecting with that enabled.

Please correct me if I'm wrong, but I suspect the lingering task is caused by xo making an HTTPS request to the pool master for rrd updates but the request fails due to a cert error and the task isn't being cancelled and instead timing out after 24 hours as Olivier has noted in past threads.

To try to resolve this, today I created a snapshot at 1:11pm PDT and booted up a fast-clone of that snapshot so I am not left without a paddle if I were to break my xo during the update. I then used the build tool's update feature to pull from master and rebuild to see if it would make a difference.

Now, when I try to connect my xo to my pool master with Unauthorized Certificates disabled, I get this error: self signed certificate in certificate chain and xo is not able to connect to the pool. If I connect with Unauthorized Certificates enabled, I am able to get stats without the lingering rrd updates tasks, but I still want to be able to connet with Unauthorized Certificates disabled.

I reviewed the xo logs and saw Hostname/IP does not match certificate's altnames: IP: 10.0.60.12 is not in the cert's list:. My master's management IP is 10.0.10.12 and the 10.0.60.12 is the address on my storage network where my storage server is located. Full disclosure, I originally mis-read the IP address as the management address, so I decided to create a new certificate following the official 8.2 instructions from Citrix with 4096 bit key, sha256 signature algorithm, 398 day lifetime, and I was sure to add both fqdns and both IP addresses as Subject Alternate Names to be sure I covered all bases.

In retrospect, the first instance of that Hostname/IP... error was over an hour before I performed the update today, so it's not new after the update. I have the original certificate which I can restore if requested.

Did I miss a step when configuring xcp-ng hosts and my xo to use the certs from my internal, self-signed CA? I have used update-ca-certificates on my xo vm to add my root ca certificate (actually, this was done and tested on the vm from which my xo vm was fast-cloned), but do I also need add my root ca certificate to each xcp-ng host as well or is that not required since the full cert chain is included in /etc/xensource/xapi-ssl.pem?

Thank you in advance for the assistance for any guidance/assistance.

@olivierlambert Now that you mention it, I do recall reading that we should always be on Master if I have issues -- I will commit that to memory and be sure to use that as a first-step

It's not often that we get to speak directly to the developers of quality projects and less often still that they are polite and helpful. Thank you for the assistance!

@olivierlambert I just confirmed that I am indeed tracking master. I am using a third party tool that basically follows the steps in your doc. I understand you have no obligation to support any procedure other than the official doc, but I do appreciate your suggestions and insights -- this is a home system, nothing is running in "production" per se, but I like when I can get things workin as expected

I have updated to latest commit on master and rebuilt, I can confirm that my continuous replication to my local storage is working. Once that completes, I'll try replication to my NFS shared SR as well. Assuming that works, I'll try my metadata backup to my NFS Remotes, then finally I'll share my results here.

Thanks again!

@olivierlambert I am using XO from sources. I haven't yet tried Tony's suggestions, but because a lot can change when tracking the master branch, I'll try updating then share my results.

@tony The MAC address of the NIC Bond seemed funny to me too, but it has been that way every time I have created a bond interface with xcp-ng—and I've made bone-headed mistakes and had to start over a few times over the years as I've learned the platform . It has seemed to work like that so I didn't bother to look into it further.

I had also tried to set up a custom xapi-ssl.pem certificate from memory on the host that happens to be my pool master at the moment and it's possible I made a mistake since I didn't take notes last time and it's possible that this could be causing some issues as well.

All of this has happened because I didn't know how (and frankly still don't know for sure yet) to properly back up a host and I needed to shift hard drives around through my pool and NAS to make bet use of my resources. I ended up thrashing through vlans, vifs, and vdbs using xe because they were tied to the host that was lost without a backup >_<

I know, I'm piling complications onto this topic, but I'm just trying to focus on one thing at a time right now..

I haven't tried the replication before NIC bonding, but I could certainly give that a try. I will probably try to do a delta first before playing with nic bonding.

Thanks again for the help.