CBT: the thread to centralize your feedback

olivierlambert

Could you try to migrate with xe and see if you have the same issue?

flakpyro

For sure, i ran:

 xe vm-migrate uuid=a14f0ad0-854f-b7a8-de5c-88056100b6c6 host-uuid=c354a202-3b30-486b-9645-2fd713dee85f

To move the VM from host 1 to host 2....

Doing it this way i noticed checking the CBT log file does not result in all zeros being output.

[10:00 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 087ad136-f31b-4d7c-9271-7c926fd51089.cbtlog 
fe6e3edd-4d63-4005-b0f3-932f5f34e036

For fun i then moved the VM back from Host 2 to host 1 and again, the cbtlog file seems to be intact:

[10:02 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]# cbt-util get -c -n 087ad136-f31b-4d7c-9271-7c926fd51089.cbtlog 
fe6e3edd-4d63-4005-b0f3-932f5f34e036

After all this migrating i then ran a job which ran fine and without any errors about not being able to do a delta.

So it seems like it works fine via xe CLI

Update:
After the backup ran properly and generated a new CBT log file i then moved it back and forth between hosts again using the CLI. And the cbtlog file seems to stay in tact again when checking using cbt-util. When i do this with XOA the result from cbtutil is all zeros.

olivierlambert

Okay so back to being an XO issue regarding migration, like if XCP thought it was migrating storage. Ping @julien-f or @MathieuRA

@flakpyro can you provide screenshot on how you migrate with XOA?

flakpyro

@olivierlambert

Here is a screenshot of how i am doing the migration in XOA: moving from host 2 to host 1, leaving the SR drop down empty.

olivierlambert

And when doing that, you are losing the CBT thing (like it's all 00000), right?

And if you unselect everything but the destination host, is it behaving the same?

flakpyro

@olivierlambert
We're making progress i think!

Correct letting the migration run with those settings results in 0000 when running the cbt-check command.

I tried removing the migration network and ran a migration with the following settings:

Screenshot 2024-12-18 at 2.28.44 PM.png

Before migration:

[14:27 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]#  cbt-util get -c -n 7560326c-8b15-4c58-841f-6a8f962a7d28.cbtlog 
fe6e3edd-4d63-4005-b0f3-932f5f34e036

And after migration:

[14:27 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]#  cbt-util get -c -n 7560326c-8b15-4c58-841f-6a8f962a7d28.cbtlog 
fe6e3edd-4d63-4005-b0f3-932f5f34e036

If i select a default migration network and run the same migration:

[14:31 xcpng-test-01 45e457aa-16f8-41e0-d03d-8201e69638be]#  cbt-util get -c -n 7560326c-8b15-4c58-841f-6a8f962a7d28.cbtlog 
00000000-0000-0000-0000-000000000000

I think we're getting somewhere now! I have the migration network on both the test and DR pools. This used to be our "Vmotion" network back when we ran vsphere and i decided to continue using it to keep migration traffic on an isolated secure vlan.

In fact these Veeam VMs are not even being used anymore they exist in our test lab as VMs to mess around with for things like this.

olivierlambert

So selecting the migration network is like triggering something in XO that makes XCP-ng moving the VDI like it was a storage migration (resetting the CBT status).

Ping @MathieuRA and/or @julien-f

flakpyro

@olivierlambert Glad we're getting to the bottom of this!

Out of curiosity is having an isolated migration network only available to the XCP-NG hosts considered best practice with XCP-NG? It was with VMware to keep VMotion traffic on its own subnet and since the VLAN was already created on our switches i decided to keep with that setup. Ideally we can get this fixed either way, I'm just curious if I'm doing something considered strange?

olivierlambert

No it's not strange at all. It's not a bad practice neither

Now, I'd like to check something with xe: migrate the VM but with the migration network selected. To see if this resets CBT or not.

flakpyro

@olivierlambert
Im on it! However after searching the XCP-NG docs as well as the XenServer docs i can't see to find how to specify a migration network using xe from the cli. Are you able to provide me the flag i need to use?

olivierlambert

I don't remember the command but @MathieuRA should be able to tell you which call we do to the XAPI when we add a migration network.

flakpyro

@olivierlambert @MathieuRA once you are able to provide me xe migrate flag to specify a migration network i will test this ASAP. I think we're really close to getting to the bottom of this issue!

rtjdamen

Hi All,

First of all best wished to you all for 2025! I have just deployed the latest build to do some testing on the one remaining issue we have with cbt backups, we were still facing full backups on some vms, this is expected to happen because cbt is not activated fast enough on some vdi’s, i will update this post once it completed some test runs to let u know if this build resolves it (there is a fix inside this build that should fix it).

Robin

olivierlambert

Happy new year and thank you very much for the feedback provided in here

rtjdamen

@olivierlambert my pleasure, good to be a part of it.

Good news, this bug seems to be resolved!

Hope we can fix the migration bug as well!

flakpyro

I think we have a pretty good idea of the cause now, It seems to be related to having a migration network specific at the pool level.

I think we are closer than ever to having this worked out and should help a lot of us using a dedicated migration network. (As was best practice in Vmware land) What are the next steps we need to take?

olivierlambert

We need to wait for the largest part of the team to get back from vacation on Monday

MathieuRA

Hi @flakpyro
You can do xe help vm-migrate to see all available parameters and a small description.

BTW, in XO, if a network is specified for the migration, we call vm.migrate_send otherwise vm.pool_migrate.
vm.migrate_send also migrates VM's VIFs and VDIs.

Questions for the XCP team:

What happens if you do a vm.migrate_send but the destination SR for the VDIs is the same?
Is there a way to call vm.pool_migrate using a specific network?

flakpyro

@MathieuRA

Thanks for the tip!

Looking at the output:

command name            : vm-migrate
        reqd params     : 
        optional params : live, host, host-uuid, remote-master, remote-username, remote-password, remote-network, force, copy, compress, vif:, vdi:, <vm-selectors>

Ir does not appear there is a way for me to specify a migration network using the vm-migrate command?

It sounds to me like vm.migrate_send is causing CBT to be reset while vm.pool_migrate is leaving it intact? The difference between a migration that is known to be kept within a pool vs one that could potentially be migrating a VM anywhere?

olivierlambert

Adding @psafont in the loop because it seems we should use pool_migrate with a network migration and/or migrate_send not resetting the CBT.