@florent Testing a storage migration i do see CBT get disabled and reset during the process which is expected! I do notice it leaves the .cbtlog file on the old SR after the storage migration is complete but that's easy enough to clean up manually.
The issue i posted above however is just a VM migration from host to host on a shared NFS SR, the SR the VM is on is not changing.
@florent thanks for checking into this as we'd love to be able to use this feature. If you need me to test anything or provide any additional logs/info about our environment let me know!
@olivierlambert So i guess the next thing we need to do is have someone also running 8.3 test this using an NFS SR?
@rtjdamen Both pools are on 8.3 with all the latest updates.
I did find this PR on github and wonder if it may be related: https://github.com/vatesfr/xen-orchestra/pull/8127 but not sure why it would only happen after a migration....
Looks like it does this if the VM is powered off as well. Im really not sure what else to try since this is happening on 2 different pools for us.
I may need to end up submitting a ticket with Vates for them to get to the bottom of it.
This is a completely different 5 host pool backed by a Pure storage array with SRs mounted via NFSv3, migrating a VM between hosts results in the same issue.
Before migration:
[01:41 xcpng-prd-03 b04d9910-8671-750f-050e-8b55c64fbede]# cbt-util get -c -n 83035854-b5a9-4f7e-869f-abe43ddc658d.cbtlog
e28065ff-342f-4eae-a910-b91842dd39ca
After migration
[01:41 xcpng-prd-03 b04d9910-8671-750f-050e-8b55c64fbede]# cbt-util get -c -n 83035854-b5a9-4f7e-869f-abe43ddc658d.cbtlog
00000000-0000-0000-0000-000000000000
I dont think i have anything "custom" running that would be causing this so no idea why this is happening but its happening on multiple pools for us.
@rtjdamen Hmm very strange.
The only thing i can think of is that this maybe due to the fact these VMs were imported from VMware.
Next week i can try creating a brand new NFSv3 SR (Since NFS4 has created issues in the past) as well as a new clean install VM that was not imported from VMware and see if the issue persists.
@rtjdamen interesting, this is with iSCSI (block) or with an NFS SR?
@olivierlambert Im making progress getting to the bottom of this thanks to some documentation from XenServer about using cbt-util.
You can use the cbt-util utility, which helps establish chain relationship. If the VDI snapshots are not linked by changed block metadata, you get errors like “SR_BACKEND_FAILURE_460”, “Failed to calculate changed blocks for given VDIs”, and “Source and target VDI are unrelated”.
Example usage of cbt-util:
cbt-util get –c –n <name of cbt log file>
The -c option prints the child log file UUID.
I cleared all CBT snapshots from my test VMs and run a full backup on each VM. Then ensured the CBT chain was consistent using cbt-util, the output was:
[14:22 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 867063fc-4d86-420a-9ad2-dfe1749ecbc1.cbtlog
1950d6a3-c6a9-4b0c-b79f-068dd44479cc
After the backup was complete i then migrated the VM to the second host in the pool and ran the same command from both hosts:
[14:26 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 867063fc-4d86-420a-9ad2-dfe1749ecbc1.cbtlog
00000000-0000-0000-0000-000000000000
And from the second host:
[14:26 xcpng-test-02 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 867063fc-4d86-420a-9ad2-dfe1749ecbc1.cbtlog
00000000-0000-0000-0000-000000000000
That clearly is the problem right there, question is, what is causing that to happen?
After running another full the zero'd out cbtlog file is removed and a new one is created which will work fine until the VM is migrated again:
[14:39 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 1eefb7bf-9dc3-4830-8352-441a77412576.cbtlog
1950d6a3-c6a9-4b0c-b79f-068dd44479cc
SMLog output on the test pool looks the same as production pool after a manual VM migration:
I did also double check that the VM UUID does not change after the migration.
Nov 15 13:59:40 xcpng-test-01 SM: [277865] lock: opening lock file /var/lock/sm/8b0ee29e-7cbe-4e15-bd13-330a974fde2a/cbtlog
Nov 15 13:59:40 xcpng-test-01 SM: [277865] lock: acquired /var/lock/sm/8b0ee29e-7cbe-4e15-bd13-330a974fde2a/cbtlog
Nov 15 13:59:40 xcpng-test-01 SM: [277865] ['/usr/sbin/cbt-util', 'get', '-n', '/var/run/sr-mount/45e457aa-16f8-41e0-d03d-8201e69638be/8b0ee29e-7cbe-4e15-bd13-330a974fde2a.cbtlog', '-c']
Nov 15 13:59:40 xcpng-test-01 SM: [277865] pread SUCCESS
Nov 15 13:59:40 xcpng-test-01 SM: [277865] lock: released /var/lock/sm/8b0ee29e-7cbe-4e15-bd13-330a974fde2a/cbtlog
Nov 15 13:59:40 xcpng-test-01 SM: [277865] Raising exception [460, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]]
Nov 15 13:59:40 xcpng-test-01 SM: [277865] ***** generic exception: vdi_list_changed_blocks: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 15 13:59:40 xcpng-test-01 SM: [277865] return self._run_locked(sr)
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
Nov 15 13:59:40 xcpng-test-01 SM: [277865] rv = self._run(sr, target)
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/SRCommand.py", line 326, in _run
Nov 15 13:59:40 xcpng-test-01 SM: [277865] return target.list_changed_blocks()
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/VDI.py", line 757, in list_changed_blocks
Nov 15 13:59:40 xcpng-test-01 SM: [277865] "Source and target VDI are unrelated")
Nov 15 13:59:40 xcpng-test-01 SM: [277865]
Nov 15 13:59:40 xcpng-test-01 SM: [277865] ***** NFS VHD: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/SRCommand.py", line 385, in run
Nov 15 13:59:40 xcpng-test-01 SM: [277865] ret = cmd.run(sr)
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 15 13:59:40 xcpng-test-01 SM: [277865] return self._run_locked(sr)
Nov 15 13:59:40 xcpng-test-01 SM: [277865] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
--
Nov 15 13:59:45 xcpng-test-01 SM: [278274] lock: opening lock file /var/lock/sm/fa7929aa-a39c-437d-9787-5218e9bcbc1a/cbtlog
Nov 15 13:59:45 xcpng-test-01 SM: [278274] lock: acquired /var/lock/sm/fa7929aa-a39c-437d-9787-5218e9bcbc1a/cbtlog
Nov 15 13:59:45 xcpng-test-01 SM: [278274] ['/usr/sbin/cbt-util', 'get', '-n', '/var/run/sr-mount/45e457aa-16f8-41e0-d03d-8201e69638be/fa7929aa-a39c-437d-9787-5218e9bcbc1a.cbtlog', '-c']
Nov 15 13:59:45 xcpng-test-01 SM: [278274] pread SUCCESS
Nov 15 13:59:45 xcpng-test-01 SM: [278274] lock: released /var/lock/sm/fa7929aa-a39c-437d-9787-5218e9bcbc1a/cbtlog
Nov 15 13:59:45 xcpng-test-01 SM: [278274] Raising exception [460, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]]
Nov 15 13:59:45 xcpng-test-01 SM: [278274] ***** generic exception: vdi_list_changed_blocks: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 15 13:59:45 xcpng-test-01 SM: [278274] return self._run_locked(sr)
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
Nov 15 13:59:45 xcpng-test-01 SM: [278274] rv = self._run(sr, target)
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/SRCommand.py", line 326, in _run
Nov 15 13:59:45 xcpng-test-01 SM: [278274] return target.list_changed_blocks()
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/VDI.py", line 757, in list_changed_blocks
Nov 15 13:59:45 xcpng-test-01 SM: [278274] "Source and target VDI are unrelated")
Nov 15 13:59:45 xcpng-test-01 SM: [278274]
Nov 15 13:59:45 xcpng-test-01 SM: [278274] ***** NFS VHD: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/SRCommand.py", line 385, in run
Nov 15 13:59:45 xcpng-test-01 SM: [278274] ret = cmd.run(sr)
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 15 13:59:45 xcpng-test-01 SM: [278274] return self._run_locked(sr)
Nov 15 13:59:45 xcpng-test-01 SM: [278274] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
@olivierlambert So today i installed the latest round of updates on the test pool which moved all the VMs back and forth during a rolling pool update. I then let everything sit for a couple hours then ran the backup job and this time it did not throw any errors. So thats even more confusing.
Perhaps its because i am kicking off a backup job immediately after migrating the VMs? As a test i am going to move them around again now, wait an hour then attempt to run the job.
Edit: Waiting did not seem to help. Running the job manually again resulted in a full being run again with the same
Can't do delta with this vdi, transfer will be a full
Can't do delta, will try to get a full stream
@gduperrey Installed on a 2 host AMD based test pool, as well as our 5 host Intel based production pool without issue using rolling pool update. Everything migrated, updated then migrated back perfectly.
Also installed on my home server without issue.
@olivierlambert Yeah we did confirm the UUID does not change. These are VMs that were imported from VMware using the XOA wizard if that makes any difference.
The only other pool i have i could try on would be our production pool which is using CBT but is not removing the snapshot.
However if i migrate a VM between hosts on that pool and run a backup job i see this on the pool masters SMLog
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] Size of bitmap: 491520
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] lock: acquired /var/lock/sm/c63f7214-c1ef-4fc3-b174-cb78dffcbaa6/cbtlog
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] ['/usr/sbin/cbt-util', 'get', '-n', '/var/run/sr-mount/16e4ecd2-583e-e2a0-5d3d-8e53ae9c1429/c63f7214-c1ef-4fc3-b174-cb78dffcbaa6.cbtlog', '-c']
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] pread SUCCESS
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] lock: released /var/lock/sm/c63f7214-c1ef-4fc3-b174-cb78dffcbaa6/cbtlog
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] Raising exception [460, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]]
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] ***** generic exception: vdi_list_changed_blocks: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] return self._run_locked(sr)
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] rv = self._run(sr, target)
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/SRCommand.py", line 326, in _run
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] return target.list_changed_blocks()
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/VDI.py", line 757, in list_changed_blocks
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] "Source and target VDI are unrelated")
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818]
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] ***** NFS VHD: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/SRCommand.py", line 385, in run
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] ret = cmd.run(sr)
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] return self._run_locked(sr)
Nov 14 21:06:50 xcpng-prd-03 SM: [3980818] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
--
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] Size of bitmap: 25165824
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] lock: acquired /var/lock/sm/90202d5b-8016-49a3-92dd-4790a2edaa41/cbtlog
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] ['/usr/sbin/cbt-util', 'get', '-n', '/var/run/sr-mount/16e4ecd2-583e-e2a0-5d3d-8e53ae9c1429/90202d5b-8016-49a3-92dd-4790a2edaa41.cbtlog', '-c']
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] pread SUCCESS
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] lock: released /var/lock/sm/90202d5b-8016-49a3-92dd-4790a2edaa41/cbtlog
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] Raising exception [460, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]]
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] ***** generic exception: vdi_list_changed_blocks: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] return self._run_locked(sr)
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] rv = self._run(sr, target)
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/SRCommand.py", line 326, in _run
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] return target.list_changed_blocks()
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/VDI.py", line 757, in list_changed_blocks
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] "Source and target VDI are unrelated")
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883]
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] ***** NFS VHD: EXCEPTION <class 'xs_errors.SROSError'>, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/SRCommand.py", line 385, in run
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] ret = cmd.run(sr)
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/SRCommand.py", line 111, in run
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] return self._run_locked(sr)
Nov 14 21:07:05 xcpng-prd-03 SM: [3981883] File "/opt/xensource/sm/SRCommand.py", line 161, in _run_locked
This goes away after the the initial backup run after the VM migration and all runs after result in nothing in the SMLog. Because i am not removing the snapshot the backup still runs as a delta however this leads me to believe if i enabled snapshot delete on that pool i would run into the same issue and a full backup would run instead.
I'm not sure what i would be doing that's unique to cause this. TestLab is NFS4 on a TrueNAS and Production is NFS3 on a Pure X20 array.
I will soon be able to test this on our DR site pool but we are currently waiting on hardware for that to arrive.
@olivierlambert Unfortunately after updating to 5.100.2 (XOA and Proxies) i am still running into XOA backups falling back to full after preforming a VM migration between two hosts within a pool when snapshot deletion is enabled. This doesnt happen if the snapshot is retained.
My test setup:
Servers: 2 x HP DL325 Gen 10 AMD Epyc Servers
Storage: TrueNAS Mini R running TrueNAS Scale 24.10 serving an NFS SR shared between both servers.
All VMs running on the same shared SR on the TrueNAS.
Scenario:
VM 1 Migrated from Host A to Host B
VM 2 Migrated from Host B to Host A
Running a CBT with Snapshot delete backup after migration the VM between hosts while they remain on the same shared SR results in the error:
"Can't do delta with this vdi, transfer will be a full"
"Can't do delta, will try to get a full stream"
Full log and screenshot of the error:
2024-11-14T19_20_40.383Z - backup NG.json.txt
@stormi Updated a test machine running only couple VMs. Everything installed fine and rebooted without issue.
Machine is:
Intel Xeon E-2336
SuperMicro board.
One VM happens to be windows based with an Nvidia GPU passed though to it running Blue Iris using the MSR fixed found elsewhere on these forums, fix continues to work with this version of Xen. 
@rtjdamen @florent Be very curious to test this once it hits XOA.
The lab i am testing this a two host pool with each server having a 24 Core Epyc CPU and 256 GB of ram. There are only around 4 VMs running on this test environment currently and all are low load VMs.
Is it still something that mostly occurs with file based SRs? I never did get a chance to setup ISCSI and test with it instead of NFS.
Once this becomes stable there i definitely plan to switch to this backup method in production!
I'm trying to make this happen on my test VM. Are you saying your need to promote the VM to a domain controller to kick off this bug? I created my VM using the Server 2022 template which did not setup a vTPM or enable secure boot. I'm wondering if that's at all related?
@rtjdamen Thats great news, with both block and file based SRs?
I created a VM using the RTM release ISO and installed the latest Citrix tools (9.4), i've left this VM running for 5 days now and i don't see any runaway conhost.exe processes, granted the VM is just sitting there doing nothing.
@rtjdamen this test pool is running on TrueNAS, so i could configure iSCSI. Our production is on NFS so i tried to keep test close to the same as far as storage is concerned. We currently use NBD + CBT without the snapshot delete function in production and it works well via NFS. Will continue to keep testing as updates roll out and look forward to when this is completely stable. If there's anything anyone at Vates needs log or test wise i'm happy to help!