Just as an update on this. Did get the migrations working. Still haven't gotten the backups working. Pool is still stable otherwise. Have managed to get backups by using Veeam so it's not to much of a problem.

@poddingue said: While you experiment, do the failing VMs have something in common? The same source SR, for example, or a mirror target keeping a different retention than the source. There are two backup jobs that are failing. One from Ark (local) to Synology, and one from Ark to S3. It seems to be the same VMs failing each night, but interestingly, there are some that fail to Synology but succeed to S3 and vice-versa. There are 9 VMs that fail for both. They do not all share the same SR. The backups on Ark are coming from several delta jobs that run daily, weekly, and monthly. Retention are different for all those.

@zorro So with purge off, Veeam stayed clean for a few nights, which fits what I saw in the lab test. If I read the docs right, the cost is that XO's reference snapshot keeps its data on the SR again, which you'd notice more on thick-provisioned storage: https://docs.xen-orchestra.com/backups-and-dr/backup-types/incremental_backups#purge-snapshot-data-cbt. Storage hasn't said yet whether purge breaking another tool's chain is expected, so I'd still call this a workaround rather than the answer. If a CBT warning comes back when you add the other VMs, the Veeam task id and the time would probably help them. And keeping both makes sense to me, since Veeam doesn't do the cross-site replication you need XO for.

Thanks @jr-m4, reporting the small ones is useful too. From what I can see in the code, the VM view of the restore page has that refresh button and the Metadata view only has a link back, so I think your screenshots show a real gap and not a glitch on your side. I honestly don't know whether XO 5 still takes this kind of small UI fix now that most of the work goes into XO 6, so I'll leave that to @Team-XO-Frontend to decide.

@Velocity1192 If you can wait a little, we are atively working on the new XO6 backup job uI, that will come with some changes : one schedule per job, but with conditionnal actions You will run an incremental job, and will be able to define that the schedule of sunday is a full,and even run a healthcheck on it The ETA is on 6.11 ( next month release )

Good to hear block mode did it. If I'm reading XO's troubleshooting page right, that 5-minute wait is a setting you can raise: httpInactivityTimeout under [xapiOptions] in a file like /etc/xo-server/config.httpInactivityTimeout.toml, then a restart of xo-server (https://docs.xen-orchestra.com/backups-and-dr/backup_troubleshooting#error-http-connection-has-timed-out). That path is the XOA one, and I don't know where the ronivay installer keeps its config, so check that first. The page calls it a workaround and says the slow host is the real thing to look at, which could fit your SMR-disk theory, but I haven't tried anything close to a 3 TB qcow2 delta myself. For the close() error, unless I'm misreading the code, XO closes the source after the failed start and only logs a warning before falling back to a full, so I'd guess it's noise from the timeout and not a second bug. The stuck tapdisks look more like one for @Team-Storage, and I don't know if a keep-alive during the delta computation is planned.

@poddingue Thank you!

OK, marking this solved... It was the phantom reboots causing issues because the tools were not fully installed, only thing I can't understand is why only 2 out of 3 Windows servers would quit. Any way, got the Xen Tools updated and a couple reboots later, everything is working again. Also I can now report that a bug that was fixed maybe a year ago is confirmed to work in my system. The bug was backing up a Windows VM that had a lot of empty space in the drive (over 250GB free) and turning on compression. It would time out counting zeros because it thought the process wasn't doing work. Both of these VMs are now compressing with zstd, one of them is substantially smaller now, the other only shrunk about 20GB. Because of the way Windows uses the drive, these are all full backups. The differential never seemed to be small because Windows does a lot of changes on the drive (there is a post about this somewhere in here from Olivier).

@poddingue Thank you!

@ma_ron said: Hi, Xen Orchestra, Community 51bb8 - back up logs and statuses are missing in the GUI. backups are done as seen exist available at the restore page. what can be the issue? Hello @ma_ron - That commit is 30 commits behind the current master commit. I suggest updating and trying again.

Glad the ticket sorted it, @jacob.becker, and thanks for coming back to say so. I don't know whether a patch for the stable channel is planned before 6.9 moves there. If someone else on stable lands here with the same Body Timeout Error, opening a ticket like you did seems to be the quickest route for now.

Thanks for coming back with that. Commit 8c2f3 is PR #10518 (https://github.com/vatesfr/xen-orchestra/pull/10518), and the changelog lists it under XO 6.9.1 from 5 October, so XOA users get it with 6.9.1, on the latest channel for now if I read the release channels right.

@johnnezero You can use the docs directly in /rest/v0/docs. You just have to connect and fill in the uuid you need. It will work

Wonderful !

Just recieved another update from veeam... Regarding the general earlier errors: Warning : Failed to use CBT: [Task e75c4247-2ee1-7b51-088c-b970ace60f3d (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated] Our QA team did hear back from Vates support, who says its tied to a recent bug they've found with blktap calls: https://github.com/xapi-project/blktap/pull/444 Our QA engineer hasn't heard back yet on ETA when it will be pushed out in a general release yet though

@poddingue Thank you!!!