Lots of Tasks: Xapi#getResource /rrd_updates (on **masterhost**) 0%

bignick8t3

Hi All,

Been using XOA community edition and XCP-NG for a while now and all has been plain sailing.

Yesterday I patched all three XCP-NG hosts up to date starting with the master of course, once this was done I updated XOA as well.

I now see lots of these tasks for the host that is running as the master, if I change the master the tasks follow to the new master etc.

I did see the following post:
https://xcp-ng.org/forum/topic/4926/thousands-of-tasks-xapi-getresource-rrd_updates-on-xcp-ng-01-0

I don't think this applies to me as all three XCP-NG hosts have their management interface in the same network as XOA.

If I restart the toolstack on the XCP-NG host acting as the master these tasks go away.

Thanks all,

Nick

olivierlambert

Hi!

Are you on latest or stable on your XOA?

bignick8t3

@olivierlambert

I've been using the following to build/update XOA:
https://github.com/Jarli01/xenorchestra_updater

The current switch I used is "lts" and this gives me xo-server 5.82.1 & xo-web 5.87.0 from the "About" within XOA.

Not sure if that answers your question but I hope so.

olivierlambert

So it's not XOA XOA means Xen Orchestra virtual Appliance that you can find on https://xen-orchestra.com

It's installing from the sources. Anyway, @julien-f is working on a fix, we'll let you know how to test it as soon it's available.

julien-f

@bignick8t3 Thanks for your report, I've just pushed a fixed on master, please test and keep me posted

bignick8t3

@olivierlambert

Oops my apologies on my terminology!
I shall make a mental note of that one.

I've also had an issue crop up with regards to my backups, would that be better in another thread?
Again has happened after me patching everything.

Thanks again.

You can ignore this as the fix that @julien-f has implemented looks to have fixed this also.

bignick8t3

@julien-f

Looks to have done the trick, no more 100's of tasks!

I also had an issue with backups whereby they were trying to access the XCP-NG host via their storage adaptors to which the appliance had to access to.
Your fix seems to have resolved that as well as I was able to rerun the backup after your fix was applied without issue.

HeMaN

@julien-f
In my weekly report email from XOCE I noticed there were no stats available about the hosts and vms.

I took a look at the master host (with XOCE) and there are also no stats available for host or vms.
Noticed a lot of rrd-updates tasks that were running but never finished

Did a restart of the toolstack, tasks are cleared, but still no statistic data available
Did a reboot of the entire server (master host) but no difference, rrd_updates tasks also back without finishing
Updated XOCE to the latest commits of today but also no difference. No stats and rrd_updates tasks never finishing.

Is there anything I can check or do to analyse/fix?

edit:
XCP-ng:
8.20 fully patched
Dell R320 / 92GB / E5-2420 v2 / intel x710da2 10gb

XOCE :
Current branch master
Current version 5.82.4 / 5.88.0
Current commit 9ceba1d6e8b0027fe97288fdf66f8946ba69a4f6 2021-10-15 14:25:02 +0200

m4xm0rris

@heman Not sure if it helps, but I had this exact problem in my pool. My shared storage was on another network that wasn't reachable by my XO host, but as soon as I added a second VIF to the XO host so that it could talk on the storage network, the issue went away and stats were available.

olivierlambert

Same question: are they in the same network?

Danp

FWIW, there's an open issue regarding missing stats. https://github.com/vatesfr/xen-orchestra/issues/5929

dsiminiuk created this issue in vatesfr/xen-orchestra

closed No Stats as of f93e1e169 #5929

HeMaN

@danp
I think you are all 3 right.

Stats were working before I did an update last week. So we're my backups (more later).

First I received an error my backups to my nas failed. Error was it could not connect to my nas ip:443.
my nas is on my storage vlan and indeed not accessible from my xoce vm.
I also noticed that the weekly report e-mail was missing all kind of stats.

I finally fixed the backup issue by adding a third nic for the storage vlan (beside my mgmt vlan and regular data untagged lan).

My migration network is also set to the storage vlan.
I looked at the statistics today and now they are also present again.

adding the vif for the storage/migration network fixed that as well without me knowing I did

thank for your help and insights

Edit: and after clearing the rrd-updates tasks by restarting the toolstack, they were also gone and did not come back (what was this topic about )

elementalwindx

@heman can you explain in a bit of detail how you did that? Did it fix this issue? I'm still pretty new to xo. Thanks.

HeMaN

@elementalwindx do you have a separate (v)lan for storage and/or migration network? if yes, is it using dhcp?

elementalwindx

@heman

I have 2 xcp-ng servers setup in a cloud setup.

They have several 10gb and 56gb nics. The 56gb nics are from A to B between the 2 servers with no switch between. The 10gb gets them to the rest of the network.
My management nic is using static ip.
The 56gb nics are using static as well.

Where can I check if I have one setup for storage? I have no idea how to set one for storage but would certainly love that ability!

HeMaN

@elementalwindx it is rather off topic for this thread, but if you are using a storage repository that is located on a network share (e.g. nfs or samba) that is used by both hosts you have shared storage.
if that network storage is located in a separated subnet in your network and you have a separate (virtual) nic assigned to that subnet in your hosts/vms, you have a separated storage network

if you want to know more about storage you better open a separate topic, there are quite a few more people here that are a lot more knowledgeable then I am

elementalwindx

@heman I'll create a new thread there. Wish we still had the IRC chat. Is there another chat method everyone has moved to?

The only thing I have going over SMB is a SMB location setup in settings and I only use that as a place to store the XO conf and delta backups.

The continuous replications are supposed to go from XCP1 to XCP2 via their 56gb nic, but it appears that only is possible to go over the management nic which is the 10gb nic. So essentially my 56gb nics are not even really being used.

So still not sure entirely why this issue is even an issue. I've ran this setup for 2 years now and it didn't become an issue until updating XO to the latest version

Darkbeldin

@elementalwindx https://discord.gg/U9KAVkDk Migrated to discord with a bridge with IRC come and join us.

stormi

And the IRC channel is #xcp-ng on oftc.net if you prefer IRC.