@CodeMercenary Can't help you there. The only thing I use(d) is the cli on the xcp hosts themselves.
reboot or shutdown

@Danp Ah, thank you for that.

I need to restructure things a bit but I was already thinking I would do that. The issue is that this VM also runs NUT so it's the last VM running before shutting down the servers. I reduced the memory because it takes a LONG time to suspend a VM with 16GB RAM but doesn't take long to shut one down. Between the 8 to 10 minutes it takes for XCP-ng to shut down and the time it takes to suspend a VM with 16GB RAM, I don't think my batteries will last that long.

I'll have to move NUT into a leaner VM that doesn't handle backups. That's something I was thinking I would do anyway because if there was a power outage during a backup I don't think my NUT script would be able to do what it needs to do. Based on the cron job I run to make sure my xo-cli registration is good, the xo-cli stuff won't run when the VM is hammered like this.

Thanks for helping me understand why this happens and how to fix it.

@CodeMercenary I use the paid for XOA and it backups up its config to the cloud so it wouldn't be a problem for me to restore. You can easily set something like that up by backing up to a store location then having that backup to a cloud storage location.

As far as having the XO instances running Im not sure how that would work. I know with NBD storage you should be able to connect to the hard drive with multiple XO's (I think) but the problem I ran into quickly was the fact that when I made a backup using my XOA, the XO from source would see it as a backup without an associated job. If you had three XO's running with 3 different remote storage locations I don't think you'd have a problem with that. Instead you might run into the XO's seeing the 3 different backup jobs running , 2 of which, they don't know about.

I use XOA to backup to a different storage array, then I have that storage backup to the cloud. My backups aren't really up to "standard" but thats all I can do for now. My plan is to make backups to a different storage pool on the same central storage I run my VM's on, back that whole unit up to an off site unit with ZFS replication, then back that backup to the cloud with StorJ. Right now Ive just got the one backup on the same Storage server and the StorJ.

@CodeMercenary Did some tests with the A2000 and as expected, the 12GB VRAM is the biggest limitation. Used vanilla installations of Ollama and ComfyUI with no tweaking or optimization. Especially in stable diffusion, the A2000 is about three times faster compared to the P40, but that is to be expected. I have added some results below.

Stable Diffusion tests

A2000 1024x1024, batch 1, iterations 30, cfg 4.0, euler 1.4s/it 1024x1024, batch 4, iterations 30, cfg 4.0, euler 2.5s/it = 0.6s/it P40 1024x1024, batch 1, iterations 30, cfg 4.0, euler 2.8s/it 1024x1024, batch 4, iterations 30, cfg 4.0, euler 12.1s/it = 3s/it

Inference tests

A2000 qwen2.5:14b 21 token/sec qwen2.5-coder:14b 21 token/sec llama3.2:3b-Q8 50 token/sec llama3.2:3b-Q4 60 token/sec P40 qwen2.5:14b 17 token/sec qwen2.5-coder:14b 17 token/sec llama3.2:3b-Q8 40 token/sec llama3.2:3b-Q4 48 token/sec

During heavy testing, the A2000 reached 70°C and the P40 reached 60°C both with the Dell R720 set to automatic fan control.

I guess I'll partially answer my own question by pointing out that I can background the commands with & so it should run them all at once anyway. Sorry, still new to bash scripting and my Linux skills are a bit rusty but improving.

@julien-f Thank you, that's super helpful and even easier than I thought it would be.

@CodeMercenary i have seen this as well, one of our backup repos was almost out of space, this decreased the backup speed, we thought of issues on the xoa side but it was just related to the repo itself…. So it is good to check both. The 1 gig part makes sense.

A bit after this I started having trouble again. I scrubbed my delta backup sets, switched to using SMB to access the same share and so far I'm a week in with no trouble at all. Remains to be seen if the strange failures will crop up again but so far I think this is the longest I've gone with the delta backups not having some random failure. Usually, it was a backup getting stuck in Started status for a day or two then switching to Interrupted because I'd reboot the XO VM to unstick it, then one of the VM backups will fail because a VDI is attached to DOM0. I know theoretically NFS is better than SMB but for now the one that breaks least often is the better option. Of course, maybe my issue had nothing to do with NFS but for now it's felt like SMB is more reliable.

@olivierlambert Well, last night the backup completed just fine despite me taking no action.

I updated the XO to the latest commit when I got in this morning so hopefully the issue I had back in June don't come back.

@olivierlambert It was DR. I was testing DR a while ago and after running it once I disabled the backup job so these backups have just been sitting on the server. I don't think I've rebooted that server since running that backup.

@andyhhp Well, then I'm glad I spent $35 each on the 2683s while the company spent $125 each on the 2697s. Saved me $180 and it has all the same functionality but just a bit less speed.

Inexplicably, now it's decided to start working right. I didn't do anything to the existing backup job.
f8c9acb1-b4ca-4486-8dfe-b957b94055f3-image.png

I'm glad it seems to be working now but I wish I understood why it wasn't working previously. That makes me nervous that it could stop working in the future without notice.

The only real difference is that I set up a brand new set of VMs to be backed up in the same way to the same remotes using two new backup plans. I guess this is kinda solved...

@lawrencesystems Excellent, that's what I was hoping to confirm.

@olivierlambert Sorry I didn't make this clear. I have multiple servers and one of them will not be used for production VMs. That's the server that happens to have a large amount of storage. I would not store backups on the same hardware as production work and this will not be my only backup of the production data. This server just has more storage than I have a use for so I thought it would be nice to use it for storing backups for the production servers since backups with history need a lot of space.

I do have an older server that I was considering using for backups but it just doesn't have nearly the storage as the one I'm talking about. 8TB as opposed to 90TB and it can't be easily expanded storage-wise. That one would be easier to convert to TrueNAS on bare metal.

It occurred to me last night that I could create 2TB drives for the XO VM on that server and backup to local storage in the VM. Actually one of my instances of XO is running on bare metal, instead of a VM, on that 8TB server so I could use that to backup to local storage if I didn't want to go with TrueNAS but TrueNAS has some other advantages, like providing an ISO share separate from my XCP-ng servers so it likely makes sense to go that route.

I figured, for the purpose of completion, I'd document what I did to solve this problem.

Create a new VM on the target XCP-ng host Attach 6 virtual drives to the VM in the sizes of the VM I need to migrate Install Windows Server 2022 in the target VM Format the additional 5 drives Share the 5 drives over the network From source VM on ESXi Map network drives to each of the 5 drives in the target VM Copy files and folders to each drive Originally tried using RichCopyUI which uses Robocopy but it kept crashing Finally used FreeFileSync over the course of several days and nights to do an initial copy of the data Performed one final mirror of the data from the source to the target Shut down the source VM In vCenter, detach all but the boot drive from the source VM Migrate the VM from ESXi to XCP-ng using XO Attach drive files from target VM to migrated VM Boot migrated VM, it had no network access and the CD wasn't working but a couple reboots fixed it. Only 2 of the 5 extra drives appeared. Two attached automatically but 3 weren't even visible in Disk Manager. Install XCP-ng guest tools and reboot Extra drives now working inside the VM Had to fight a little with network settings and my DHCP server to get the same IP address assigned Did some testing, had to tweak a few things in some software but everything seems to be working normally again without too much work Shut down the old host running ESXi - one step closer to severing ties with VMware and also made my server room quieter and cooler

@florent It is a very old host. In addition to moving to XCP to get away from VMware, we're upgrading to get off the oldest hosts onto a couple new hosts I just added. It's quite possible that the activity was overwhelming that rather weak old host.

@CodeMercenary To answer your questions about backups, they should all be fine, the remote is attached to XO rather than the host itself so XO can backup any VM from any host it has access to, to any remote configured in XO.