What are your needs for a backup? A full bare-metal recovery of the server/vm, or just some of the files on it?
If XCP-ng/XOA backups are too slow, why not separate the main data into a second disk mounted over iSCSI inside Windows. You'd still need to manage backups somehow though.
You mentioned open source. One option is https://www.urbackup.org/. We have been looking at it as a replacement for Acronis for bare-metal machines, but not yet made a decision.
If you just need to keep the files, you could easily make a script that creates a VSS snapshot and runs robocopy to some remote destination. I do this on several servers, and then I manage retention on the remote location separately.