@florent I reverted back to:

xo-server 5.100.0
xo-web 5.101.0
commit 26e7e

and I'm facing the same issues. If you can tell me, which logs you need, I can surely provide them.

Thanks!

@florent Apologies for my late response.

Which log exactly would you be looking for? the SMlog under /var/log on the XCP host?

@florent Hi florent, great news! I'm just wondering, which encryption you mean? On my end, I am not encrypting anything.

@florent the size of the VMs, where this problem exists, is below 2TB:

I observed this behaviour on 5 VMs, 3 of them on the same host, where 2 of them are Windows ones

• Win-DB = 100GB (on NFS-Storage with plenty of free space available)
• Win-IIS = 100GB (on NFS-Storage with plenty of free space available)
• srv02 = 55GB (sys) and 75GB (data) (both on NFS-Storage with plenty of free space available)
• vps01 = 10GB (sys) and 500GB (data) (both on NFS-Storage with plenty of free space available)
• vps02 = 10GB (sys) and 500GB (data) (both on NFS-Storage with plenty of free space available)

If I can provide anything additional, let me know.

Thanks!

@florent An observation / similiarity:

Problems occur on VMs with big disks. Could the backup run into a timeout (whyever?)?
I am backing up all my VMs using a "delta backup" on an hourly basis, all scheduled at different times within the hour (like :15 for VM1, :30 for VM2 etc.).

No idea if it helps: This issue is not on all backup jobs, only on some of them.
But I wasn't yet able to find out any similarities between those failing ones compared to those, that work fine.

@florent

My remote is a NFS share on another host, connected via 10G SFP+.
The option "Store backup as multiple data blocks instead of a whole VHD file" is not in use. This is my remote settings in XO:

@florent said in Backup through XO fails (VHD check error etc.):

@tanjix as long a sthe backup doesn't run till the end, it won't clean the repo

Have you got any performance issue on you XO ( like memory issue ) since the backup worker exits without any information ?

Hello @florent,

no, there's plenty of CPU/RAM resources available. Hope that helps?

If you want me to provide additional info, let me know.

Thanks and best wishes!

Hi Guys,

since a few days, backup jobs on some VMs fail - all with different errors. It has worked before since ages and since some days - out of the blue -

The corresponding full log is this one:

2022-10-13T08_30_00.017Z - backup NG.txt

Another VM's log:

2022-10-13T15_00_00.030Z - backup NG.txt

I am using XO from the sources, xo-server 5.103.1, xo-web 5.104.0, commit 42a97.

Any help would be appreciated!

Thanks a lot and best wishes!

Thumbs up on this, have been facing the same issue today.

Hi Olivier,

thanks, but as I described initially, those tasks are not disappearing and summing up after a while.

Did you also consider my report --> here?

Thanks!

Any news available?

@olivierlambert correct

@olivierlambert said in Stuck tasks within XO (rrd_updates)?:

xe host-param-get param-name=address uuid=

Sure, that one is returning one ip address, the public one I am talking about.

@olivierlambert What do you mean with "internally"? There is only one IP in use in my environment, which is the public IP. So yes, it's the same IP address.

@olivierlambert What are you referring to? What should I do? There is no NAT, private network etc involved. XO connected to the XCP-NG host (pool master) via its public hostname (for which the SSL certificate is issued) and NOT with its ip address only.

Thanks!

@olivierlambert So, how could I get rid of these error messages and the fact, that I have plenty of these tasks running after a while?
I guess, restarting the toolstack every hour is not the preferred way...

Thanks!

@olivierlambert is there any update on this issue available?

Thanks!

@olivierlambert

Maybe I found the problem.
In XO's log section I found an entry like:

Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list:

The detailed log says at the bottom:

"code": "ERR_TLS_CERT_ALTNAME_INVALID",
"url": "https://a.b.c.d/rrd_updates?cf=AVERAGE&host=true&interval=5&json=true&start=1611607291&session_id=OpaqueRef%3A7601e913-a96c-419e-a5ab-be65255ab3d7&task_id=OpaqueRef%3Aecd831ec-bd4f-4a40-82c4-f15059ffb377",
"message": "Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list: ",

a.b.c.d is in this case the ip address of my pool master.

I installed TLS certificates (a wildcard certificate, not self signed but from a trusted authority), so that all of my hosts are reachable ssl-secured (done that through XO --> Home --> Hosts --> <host> --> Advanced and there at the bottom).

Of course, this only works if requests are made with the hostname and not with the ip address.

If I open the URL from above and replace the ip address through the hostname of the pool master, then it looks like it works.

So, how can this be fixed that these rrd_things make their calls with the hostname instead of the ip address?
Or did I do anything wrong with the certificates?

Thanks!

@danp

@danp said in Stuck tasks within XO (rrd_updates)?:

@tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

Negative, I will do that and report later on. Thanks!