See that as a positive experience: you have learned many things

@DustinB Yeah, that's what I will try to remember to do next time. This is still worlds easier for me than trying to keep my ESXi hosts up to date was back when I used them. So very much better than that. I very rarely updated those at all.

So, a bit after I originally posted this, one of the two servers fans slowed back down and I don't know why. I only noticed it weeks later. Then this morning we had a power outage and all the servers were shut down. When I booted them back up when power was restored, the other server was running the fans at normal speed. No idea why it went back, I didn't do anything to fix it. After that reboot though, just a single fan in the server that originally didn't have that problem, is now running fast. That makes me wonder if the fan is failing so I'm looking to find some spares to keep around. Any future reboots are going to make me a bit stressed wondering if the fans will speed up again. I did install the pool patches today and that reboot didn't impact the fans, thankfully. I wish I understood what happened but if it happens again I might use this docker container to take over control of them: https://github.com/tigerblue77/Dell_iDRAC_fan_controller_Docker

@CodeMercenary Can't help you there. The only thing I use(d) is the cli on the xcp hosts themselves. reboot or shutdown

@Danp Ah, thank you for that. I need to restructure things a bit but I was already thinking I would do that. The issue is that this VM also runs NUT so it's the last VM running before shutting down the servers. I reduced the memory because it takes a LONG time to suspend a VM with 16GB RAM but doesn't take long to shut one down. Between the 8 to 10 minutes it takes for XCP-ng to shut down and the time it takes to suspend a VM with 16GB RAM, I don't think my batteries will last that long. I'll have to move NUT into a leaner VM that doesn't handle backups. That's something I was thinking I would do anyway because if there was a power outage during a backup I don't think my NUT script would be able to do what it needs to do. Based on the cron job I run to make sure my xo-cli registration is good, the xo-cli stuff won't run when the VM is hammered like this. Thanks for helping me understand why this happens and how to fix it.

@CodeMercenary I use the paid for XOA and it backups up its config to the cloud so it wouldn't be a problem for me to restore. You can easily set something like that up by backing up to a store location then having that backup to a cloud storage location. As far as having the XO instances running Im not sure how that would work. I know with NBD storage you should be able to connect to the hard drive with multiple XO's (I think) but the problem I ran into quickly was the fact that when I made a backup using my XOA, the XO from source would see it as a backup without an associated job. If you had three XO's running with 3 different remote storage locations I don't think you'd have a problem with that. Instead you might run into the XO's seeing the 3 different backup jobs running , 2 of which, they don't know about. I use XOA to backup to a different storage array, then I have that storage backup to the cloud. My backups aren't really up to "standard" but thats all I can do for now. My plan is to make backups to a different storage pool on the same central storage I run my VM's on, back that whole unit up to an off site unit with ZFS replication, then back that backup to the cloud with StorJ. Right now Ive just got the one backup on the same Storage server and the StorJ.

@CodeMercenary Did some tests with the A2000 and as expected, the 12GB VRAM is the biggest limitation. Used vanilla installations of Ollama and ComfyUI with no tweaking or optimization. Especially in stable diffusion, the A2000 is about three times faster compared to the P40, but that is to be expected. I have added some results below. Stable Diffusion tests A2000 1024x1024, batch 1, iterations 30, cfg 4.0, euler 1.4s/it 1024x1024, batch 4, iterations 30, cfg 4.0, euler 2.5s/it = 0.6s/it P40 1024x1024, batch 1, iterations 30, cfg 4.0, euler 2.8s/it 1024x1024, batch 4, iterations 30, cfg 4.0, euler 12.1s/it = 3s/it Inference tests A2000 qwen2.5:14b 21 token/sec qwen2.5-coder:14b 21 token/sec llama3.2:3b-Q8 50 token/sec llama3.2:3b-Q4 60 token/sec P40 qwen2.5:14b 17 token/sec qwen2.5-coder:14b 17 token/sec llama3.2:3b-Q8 40 token/sec llama3.2:3b-Q4 48 token/sec During heavy testing, the A2000 reached 70°C and the P40 reached 60°C both with the Dell R720 set to automatic fan control.

I guess I'll partially answer my own question by pointing out that I can background the commands with & so it should run them all at once anyway. Sorry, still new to bash scripting and my Linux skills are a bit rusty but improving.

@julien-f Thank you, that's super helpful and even easier than I thought it would be.

@CodeMercenary i have seen this as well, one of our backup repos was almost out of space, this decreased the backup speed, we thought of issues on the xoa side but it was just related to the repo itself…. So it is good to check both. The 1 gig part makes sense.

A bit after this I started having trouble again. I scrubbed my delta backup sets, switched to using SMB to access the same share and so far I'm a week in with no trouble at all. Remains to be seen if the strange failures will crop up again but so far I think this is the longest I've gone with the delta backups not having some random failure. Usually, it was a backup getting stuck in Started status for a day or two then switching to Interrupted because I'd reboot the XO VM to unstick it, then one of the VM backups will fail because a VDI is attached to DOM0. I know theoretically NFS is better than SMB but for now the one that breaks least often is the better option. Of course, maybe my issue had nothing to do with NFS but for now it's felt like SMB is more reliable.

@olivierlambert Well, last night the backup completed just fine despite me taking no action. I updated the XO to the latest commit when I got in this morning so hopefully the issue I had back in June don't come back.

@CodeMercenary I had a simillar issue to you, check the time and the date on the host. If this might be any help to you: https://xcp-ng.org/forum/topic/10687/after-an-update-the-nic-s-has-disappeared-but-still-works-somehow/10?_=1743764792775

@andyhhp Well, then I'm glad I spent $35 each on the 2683s while the company spent $125 each on the 2697s. Saved me $180 and it has all the same functionality but just a bit less speed.

Inexplicably, now it's decided to start working right. I didn't do anything to the existing backup job. [image: 1717691279000-f8c9acb1-b4ca-4486-8dfe-b957b94055f3-image.png] I'm glad it seems to be working now but I wish I understood why it wasn't working previously. That makes me nervous that it could stop working in the future without notice. The only real difference is that I set up a brand new set of VMs to be backed up in the same way to the same remotes using two new backup plans. I guess this is kinda solved...

@lawrencesystems Excellent, that's what I was hoping to confirm.