@bug-meister My opinion is that cloud-init should be used only for the most basic stuff like initial network configuration, initial user creation and credentials injection. As soon as you gain the ability to access the VM, some more robust configuration management solution like Ansible should take over.
ansible-pull mechanism is in today's day and age effectively deprecated. Just run Ansible after the VM has been provisioned to apply the configuration. If you need to call Ansible as soon as VM is provisioned without any manual invocation, you can have Terraform provision the VM and call Ansible right after that. You can even use pure Ansible for both - provisioning, with my community.general.xenserver_guest module, and configuration. You just invoke Ansible once and everything will be done in one go. The only drawback is that my module does not support cloud-init so you would need some trickery to do the stuff.