Hello,

Indeed, disabling NBD for the delta backups, they are no more VDI attached to Dom0.

My upgrade path was from 8.2.1 stable to 8.3 using the ISO installation.

@Tristis-Oris
It seems that since updating to 8.3, I also have VDI attached to control domain I cannot delete, they appear after a backup.
The only solution to forget them is to a pool reboot.

I make it the other way, I add a tag for the VM I want a health check to happen, and no tag if I don't want a health check.

Then, in the backup's schedule, I add my tag:

Hello,

Is there a way to do multiple health check without starting them one by one?

In the restore tab, if I select multiple backups with the tick box, the restore health check icon appears in the top right corner, but if I click on it, I got an error message: t is not a function.

One other way would be to create a schedule in the backup job, but it doesn't seem possible to do just a health check without first creating a new backup.

Thanks for your help,

Can you go into the BIOS menu, then Boot manager and set Windows Boot Manager as the boot option?
I think I had to do that with a Windows Server 2022 in UEFI mode imported from vmWare, but not sure, it was a while ago.

@olivierlambert
It took a while to validate it, but I can also confirm that the perf-alert plugin is the culprit.

Here are my plugin's parameters, if it can help chasing the goose:

I'm gonna try the same and will let you know.

@olivierlambert
Simpler setup on my side:

Pool 1 - 1 x ASRock Rack 1U4LW-X570/2L2T RPSU with about 10 VMs
Pool 2 - 1 x HP DL360 Gen9 with only the XO VM (Debian 11).

Each host has 2 x 10 Gb NIC in use (one for VMs/MGMT, the other one for NFS storage).

The storage is a NFS share on a QNAP NAS (except for the XO VM which is on the local storage of the DL360 host).

The 2 hosts and the NFS storage are connected on the same10Gbit Ubiquiti EdgeSwitch.

Same plugins as @14wkinnersley + audit and sdn-controller.

The issue with stuck task is only appearing on the pool 1, not on the pool 2 with only the XO VM.

@MathieuRA
OK, thanks for the answer.

Looking forward for XO6

Hello,

Maybe I've missed it, but is there a way to tag several VMs with the same tag in one operation, by doing a multi-selection in the VM window in XO?

Thanks,

@olivierlambert
Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot.

@olivierlambert
Yes, confusing, indeed
I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
I'll let you know ASAP.

@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a.

@olivierlambert
I mean if use commit 0ccfd4b, there is no issue, if I use a more recent one, like the latest 583c7, the issue is appearing.

@olivierlambert
MTU is standard @1500.
There's been no change in the network.
That issue is only there with commit newer than 0ccfd4b.

@olivierlambert
I'm not good at reading logs, but I don't see errors if I look around the time of the stuck task.

All I see is a lot of this, which doesn't seem to be an error:

Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 HTTPS "XO-IPv4-address"->:::80|task.create D:8c2a9729cdc7|taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c (uuid:fc9308de-a60e-0647-de77-5affe59f0d65) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:8c2a9729cdc7
Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 :::80||taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124062 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:0ce72cef0c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||87 |session_revalidation_func D:226b44cedeab|xapi_session] revalidating all external sessions in the local host
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 HTTPS "XO-IPv4-address"->:::80|task.create D:44cbbd1eaa07|taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 (uuid:ff57e9c2-0efe-1442-b914-f0fc03ad4298) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:44cbbd1eaa07
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 :::80||taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||123681 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:c02e53b83d06|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:d9a843aad4e0|taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 (uuid:0cf536c9-4aca-345a-21af-5b50517850a1) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d9a843aad4e0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 HTTPS "XO-IPv4-address"->:::80|task.create D:d5fe32e3d49d|taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 (uuid:136e5de2-7cff-e064-3889-1e53e94bc39a) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d5fe32e3d49d
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 HTTPS "XO-IPv4-address"->:::80|task.create D:a93c0ae92fcb|taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 (uuid:6d9f76fd-a1a4-a276-cf09-c48bda2c1bda) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a93c0ae92fcb
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 HTTPS "XO-IPv4-address"->:::80|task.create D:43d6297edb7a|taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 (uuid:d9c42476-a14c-5d22-7dad-07b4bc1beb34) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:43d6297edb7a
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:54f29cd94a60|taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c (uuid:8a409f1b-05fb-a649-dd1c-43995e9b083b) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:54f29cd94a60
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:22d06e0331f0|taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b (uuid:618903d7-0d37-4048-0d87-cc43ae2715d9) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:22d06e0331f0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124070 HTTPS "XO-IPv4-address"->:::80|task.create D:4b078a1f60f8|taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc (uuid:a853806a-c374-bd5a-4a70-0b5e653355ba) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:4b078a1f60f8
Mar 26 15:38:00 xcp-angelina xapi: [ info||124071 HTTPS "XO-IPv4-address"->:::80|task.create D:d3c88668426e|taskhelper] task Xapi#getResource /rrd_updates R:29c0be84d819 (uuid:96afdfa1-c411-1010-a02b-898f763ea2e7) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d3c88668426e
Mar 26 15:38:00 xcp-angelina xapi: [ info||124068 :::80||taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124068 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:204df6c50cf5|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 :::80||taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 :::80||taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124065 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:60d06c209765|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124064 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:45d68614eed1|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:df8c7a898e42|taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd (uuid:57e238cf-4990-e698-f92e-4db87790c025) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:df8c7a898e42
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 :::80||taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124063 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:14f097fd8f89|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 :::80||taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124066 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:96c553abff8b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124069 :::80||taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 :::80||taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124069 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:d604b8b2b8fc|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124067 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:761a0f164b4c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124072 :::80||taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124072 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:1e5f14157edd|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded

And if I track one of the stuck task by its UUID, there's only one occurrence in the logs:

Mar 26 12:37:00 xcp-angelina xapi: [ info||55610 HTTPS "XO-IPv4-address"->:::80|task.create D:a75213c5e38a|taskhelper] task Xapi#getResource /rrd_updates R:24f7cb51c61e (uuid:02585f09-7fae-fac6-8e9c-d261ba19587d) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a75213c5e38a

I can share more of the logs if needed.

@olivierlambert
I would be happy to provide any extra info you need, no worries.
The config on the host is pretty straightforward, just XCP-ng 8.2.1 with latest patch, no extra package except netdata-ui.

@olivierlambert
Yes, the IP is matching. There has been no change in IP address, and there's no routing involved, no proxy or whatever, XO and the host are in the same subnet.

@olivierlambert
Yes, toolstack restarted several times. It clears the task, and then, after a while, the task is appearing again.

I tried latest commit 583c7.
It solved restore, health check and disaster recovery certificate issues, but unfortunately, I still got the Xapi#getResource /rrd_updates tasks growing in the list.