SOLVED Thousands of tasks Xapi#getResource /rrd_updates (on xcp-ng-01) 0%
-
I woke up yesterday to 14,000+ tasks that stated the following.
Xapi#getResource /rrd_updates (on xcp-ng-01) 0% xen orchastra was also not reporting any stats on VMs.Xen-Center however was reporting everything correctly and fine.
I was unable to find anything recently via the forums or github issues since 2018.
I did not find anything meaningful in the logs other than some coalescence issues happening.In troubleshooting, i've tried many things. My last resort was to re-install XCP-NG 8.2, and also reinstall XOA from source.
I still seem to be getting the same tasks. However it seems to be hovering around 35ish.
Any ideas what this is indicating?
https://imgur.com/fgufNRw -
It won't work correctly behind a NAT (except if XOA and the host are BOTH in the same network).
That's because consoles and RRDs are fetched using the IP returned by XAPI itself (which is in general the local/private IP). Exemple:
- your host is configured with
192.168.0.10inside your home network - your XOA is running elsewhere (ie in another place)
- you are connecting your XO to your host via your public IP (eg 72.21.31.41) and you did a NAT for XAPI ports (80/443).
When XO will ask for stats on console, the URL returned by XAPI will be
192.168.0.10. So your XO will try to connect to that and… it won't work obviously
Something obviously changed in your setup, because it was always like this since ever (you can't access RRDs or console if your host is behind a NAT). Note that's something we'll "fix" by counting on XO proxies in the future.
- your host is configured with
-
Is your XAPI (host) behind a NAT?
-
@olivierlambert I can access it locally or NAT'd.
I've had it running for over a year without change.But yes. I can access it via http:xoa.domain.com and via http://192.168.15.15
I've been running it in docker, but when troubleshooting I've tried running it natively on Linux built from source with the same issue.
-
It won't work correctly behind a NAT (except if XOA and the host are BOTH in the same network).
That's because consoles and RRDs are fetched using the IP returned by XAPI itself (which is in general the local/private IP). Exemple:
- your host is configured with
192.168.0.10inside your home network - your XOA is running elsewhere (ie in another place)
- you are connecting your XO to your host via your public IP (eg 72.21.31.41) and you did a NAT for XAPI ports (80/443).
When XO will ask for stats on console, the URL returned by XAPI will be
192.168.0.10. So your XO will try to connect to that and… it won't work obviouslySomething obviously changed in your setup, because it was always like this since ever (you can't access RRDs or console if your host is behind a NAT). Note that's something we'll "fix" by counting on XO proxies in the future.
- your host is configured with
-
@olivierlambert
Should I be concerned about them?
The only fix is to put my host and XOA on the same network? -
It's just that we ask XAPI to get a resource and we can't reach it. The task will be removed after 24h.
There's no other way if you are in different network (except when we'll have XAPI proxies, but it will require to have a proxy VM in the same network, then you could have everything from a central XOA despite NATs and so on.)
-
@olivierlambert
I've been using NGINX-PROXY-MANAGER locally.
Just changed XOA to same network as host.
I still need to wait 24 hours? Or can I restart toolstack? -
You'll need to restart the toolstack on the host to clear them (or wait 24h). But if you are in the same network, you won't have new ones
-
@olivierlambert Thanks. As always you guys are great.
Thanks for this opensource project!
Tasks have cleared up.
