Great news ! Thanks @olivierlambert and @florent and let me know if you need some information on the vmware side.

Thanks for the details @florent

@florent said in What is the status/roadmap of V2V (Migrating from VMware to XCPng/XO) ?:

the newer VMFS put more lock on the files, locking the full chain of snapshot and base disks instead of locking only the active disk.
Even VMFS5 sometimes lock the full chain.

That explains why I had locking issues trying to restart the source VM on vmware after a migration test.

I'll see if I can find anything on how to use NBD with vmware.

Hi @Cyrille, thank you for working on this, it will help a lot.

This is indeed implementing the needed behavior when creating vms from templates, avoiding the use of additional ad-hoc code. For reference, I needed to call the vm.set XO JSON-RPC function after the creation with the following arguments

            rpc_args = {
                "id": vm_id,
                "memory": vm_memory,
                "memoryMin": vm_memory,
                "memoryMax": vm_memory,
                "memoryStaticMax": vm_memory,
            }
            await rpcserver.vm.set(**rpc_args)

I'll try to test the provider patch in the coming days.

Awesome ! Thanks Vates and DESY for all the work that went into this.

I'm really looking forward to use the pulumi provider when I get to the automation part of my tests (probably in a few months though).

Hi @Cyrille, thank you for working on this, it will help a lot.

This is indeed implementing the needed behavior when creating vms from templates, avoiding the use of additional ad-hoc code. For reference, I needed to call the vm.set XO JSON-RPC function after the creation with the following arguments

            rpc_args = {
                "id": vm_id,
                "memory": vm_memory,
                "memoryMin": vm_memory,
                "memoryMax": vm_memory,
                "memoryStaticMax": vm_memory,
            }
            await rpcserver.vm.set(**rpc_args)

I'll try to test the provider patch in the coming days.

Hi, after working for some time on templating and deployment with packer and pulumi, I decided to publish complete examples.

https://github.com/Kaelnor/xcp-ng-pulumi-packer-example

Hopefully, this can be useful to someone else. I tried to give enough details in the README and code, especially for issues or limitations I faced.

Of course, I did not come up with all this by myself so thanks to every one on this forum that shared their previous efforts that I could build on. I tried to cite most sources I remember using when creating this.

Feel free to comment, fork or propose improvements.

Great news ! Thanks @olivierlambert and @florent and let me know if you need some information on the vmware side.

Has there been any progress/decision on the V2V improvements for recent vsphere versions ?

Hi, I'm currently testing deployments with pulumi using packer templates.

So far the basics work as expected but I'm stuck on a setting issue that seems to affect both pulumi and terraform providers. As far as I know there is no way to set the memory as static or changing memory_min when creating a VM from a template.

The template was created with 1cpu and 2GB of RAM

The VM created from this template using pulumi was assigned 2cpus and 4GB of RAM, however this only sets memory_max

I found the following post that talks about this: https://xcp-ng.org/forum/topic/5628/xenorchestra-with-terraform

and also the folllowing github issue https://github.com/vatesfr/terraform-provider-xenorchestra/issues/211

Manually setting the memory limits after VM creation defeats the purpose of automation. I suppose that implementing those settings in the relevant providers is a core feature. In most cases, VMs need static memory limits.

In the meantime, is there any workaround that I should investigate or anything that I missed ?

EDIT: Using the JSON-RPC API of XenOrchestra, I'm able to set the memory limits after the creation of the VM. This is great but unfortunately it is a bit too "imperative" in a declarative world.
I'll publish the code when I can clean up the hellish python I wrote, but a few pointers for those interested:

• See Mickaël Baron's blog (in French, sorry!) for an exemple of working with XO JSON-RPC API: https://mickael-baron.fr/blog/2021/05/28/xo-server-websocket-jsonrcp

• The system.getMethodsInfo() RPC function will give you all available calls you can make to the server. For instance, you can sign-in with session.signInWithToken(token="XO_TOKEN") and vm.setAndRestart to change VM settings and restart it immediately after.

• You can use Pulumi's hooks: https://www.pulumi.com/docs/iac/concepts/options/hooks/

• In python, Pulumi is running in an asyncio loop already so bear that in mind: https://www.pulumi.com/docs/iac/languages-sdks/python/python-blocking-async/

slax81 created this issue in vatesfr/terraform-provider-xenorchestra

closed Dynamic memory control #211

Awesome ! Thanks Vates and DESY for all the work that went into this.

I'm really looking forward to use the pulumi provider when I get to the automation part of my tests (probably in a few months though).

@florent said in Feedback on immutability:

@rtjdamen for the immutability to be useful, the full chain must be immutable and must never be out of immutability

the merge process can't lift/ put back the immutability , and increasing synchronization between process will extend the attack surface.

immutability duration must be longer than or equal to 2 time the full backup interval -1
the retention must be strictly longer than the immutability .

for example, if you have a full backup interval of 7 a retention of 14 and immutability duration of 13 , key backup are K, delta are D. Immutable backup are in bold . unprotected chain are striked

KDDDDDDKDDDDDD worst case, only one full chain protected
KDDDDDKDDDDDDK
KDDDDKDDDDDDKD
KDDDKDDDDDDKDD
KDDKDDDDDDKDDD
KDKDDDDDDKDDDD
KKDDDDDDKDDDDD best case almost 2 full chain protected

I have not tried backups in XO yet but I'm really looking forward to test the immutability as we have it configured on all veeam backups at work.

Just to be sure, the XO immutability "agent" only does its immutability check by date right ?
Would it be possible to consider the entire backup chain related to the oldest immutable restore point instead ? This would prevent misconfigurations from the user that result in insecure backup chains.

Hi @olivierlambert @florent ,

I didn't have much time to work on this in the last weeks but I finally could dig deeper thanks to the migratekit repo.

Essentially, they are delegating all the work to nbdkit and its vddk plugin (https://gitlab.com/nbdkit/nbdkit and https://libguestfs.org/nbdkit-vddk-plugin.1.html) by spawning an external process (https://github.com/vexxhost/migratekit/blob/a08325d420733e4eb26331d87bf6ef46d8cccd7f/internal/nbdkit/builder.go#L82).

The authentication info is simply the authentication to vCenter/ESXi provided by the end-user if I'm not mistaken and the filename given to nbdkit is indeed gathered from the VirtualDeviceBackingInfo property. They are using the govmomi auto-generated library for this.

For instance, on a snapshot of one of our VMs:

You can see the property path at the top and the fileName property contains the "[datastore-name] filepath" string.

The "device[2000]" part of the path is from the list of devices attached to the VM that can also be accessed following the snapshot moref:

Migratekit is then filtering on the VirtualDisk type in the device list.

Now, the problem in this setup is that nbdkit is using VDDK directly, but the development kit cannot be redistributed without a licence agreement from Broadcom: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/8-0/virtual-disk-development-kit-programming-guide/the-virtual-disk-api-and-vsphere/developing-for-vmware-platform-products/redistributing-vddk-components.html

The user would have to download and install VDDK manually.

I hope this helps and let me know if you need more details on all this. I played a bit with pyvmomi 5+ years ago but I never used the SOAP API "manually" though.

Thanks for the details @florent

@florent said in What is the status/roadmap of V2V (Migrating from VMware to XCPng/XO) ?:

the newer VMFS put more lock on the files, locking the full chain of snapshot and base disks instead of locking only the active disk.
Even VMFS5 sometimes lock the full chain.

That explains why I had locking issues trying to restart the source VM on vmware after a migration test.

I'll see if I can find anything on how to use NBD with vmware.

@olivierlambert Thanks for the feedback.

Is the limitation only due to VMFS or both the esxi version and VMFS ? Because vsphere8 still supports VMFS5 and we could imagine a 2-step migration by manually moving VMs on a temporary datastore. However, if the issue is the API change with vsphere8 then I understand that it would be difficult indeed.

I'm sure the dev team has already explored the subject to build V2V in the first place but just in case it could help, here are the relevant veeam and vmware docs for vmdks transport modes (V2V is NBD mode if i'm not mistaken):

https://helpcenter.veeam.com/docs/backup/vsphere/transport_modes.html?ver=120
https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vddk-programming-guide/GUID-15395099-5300-4D3F-BCC3-E50DCDC954C2.html

I imagine building a viable alternative is quite a big project in itself.