@TheNorthernLight Thanks for pointing that out, that totally makes sense, I should have realized to look it up that way. I guess I just think of XCP-ng as its own product so much that I forget it's based on Xen even though I know that.

If anyone can point me to documentation on this, I would greatly appreciate it. I have looked in the docs and searched for an answer and come up empty.

I have two PE R630 and one PE R730xd. The two R630s show the IPMI data in XO but the R730xd does not. If I run ipmitool sensor from an ssh session, I get very similar data back on the 730 and 630.

This isn't critical to me, just letting you know that there are cases that functioning ipmitool on a Dell does not seem to be enough to show it.

There are some differences in the labels. The R730 has "Fan1 RPM", "Fan2 RPM" while the R630 has "Fan1A", "Fan2A", "Fan1B". Most of the other interesting fields seem to have the same names but the order is a bit different and I didn't do an exhaustive comparison.

It's very cool that you are surfacing this data now.

I was rebooting my servers due to the patches released last week and I notice that the BIOS in my server was not set to interleave the memory. Since I have symmetric memory sticks, it seems like it should be set to interleave. However, the BIOS does say that for NUMA aware OSs, that interleaving should be left to the OS. I don't recall now, and my notes don't say, if I intentionally left interleaving off because v8.2 I was installing at the time was NUMA aware.

Memory interleaving on a Dell R630: on or off?

During the last patch I turned on interleaving for my R730xd (without reading the help that says not to do it with NUMA aware OSs) and it feels faster to me, but I didn't do any real testing so that could be wishful thinking bias. It certainly doesn't seem to have hurt performance.

@nikade I did install the 97 XCP patches and that night the VM that was failing to backup was able to successfully backup. It had a 500GB secondary drive that was formatted but totally empty.

I had not yet patched my XO from source to the latest, it was maybe 24 commits behind. On Friday I updated it so I will find out on Monday if those backups continued to work but I assume they did.

Amazing timing. I have been having this issue with a VM I just set up and I came here to find out if this is a known problem, only to discover that the 97 patches I have been planning to install tonight are going to fix it.

@paco Oh, that's great information, thank you. I'd much rather reset that flag instead of constantly adjusting the fans.

I recently implemented Kopia to replace Duplicati on some systems in my network. Now that I have more than a week of backups, I have gained appreciation for the retention tags they use in the KopiaUI program.

It does a good job of indicating why this version is being retained while also showing how many copies it is maintaining at each retention level. Really handy.

I just noticed the pin to the right. That lets you pin the snapshot so it won't be removed during cleanup. That's kinda cool to be able to single one out to hang on a bit longer, maybe while investigating something I suspect went wrong after that snapshot.

I'm not shilling for Kopia, I haven't used it enough to be sure I trust it yet. I am hoping XO could blatantly steal this kind of idea for backups, especially delta backups. I often get confused about how to get the retention I want when configuring backups and when I search for answers I see other people are also confused.

I guess for the sake of completeness, this is how KopiaUI lets me configure retention:

@JamfoFL Thank you for the follow-up. Now I feel like I can install those patches. Bonus that it appears there are 23 patches now rather than the 22 I saw last week. Now I'm glad I waited, though I think the newest patch doesn't apply to my hardware anyway.

Thank you @florent. I'll do that.

Update: It worked.

Those 5 mirror backups continued to fail over the weekend.

The good news is that the normal delta backups succeeded over the weekend and didn't have any random failures I've been seeing.

I have a couple Delta backups that target the same remote. Yesterday I set up a Mirror backup between that remote and another remote. When I ran the Mirror backup yesterday, it was successful.

Today 7 of the VMs succeed in the backup but 5 of them fail with:
Missing vdi <guid> which is a base for a delta

Could this be caused by the fact that the target for the Mirror backup used to be a target for the Delta backup so it has files on it already. Do I need to purge the folder entirely? I thought maybe that would give it a head start and the reason I'm using the Mirror backup is because the Delta backup used to fail randomly on random VMs when backing up to both remotes.

How would I go about diagnosing this? I assume since the guid for the vdi is missing, it's going to be hard for me to find since I can't find something that's missing. Alternatively, maybe it's because it's currently a weird mix of old Delta backups and new Mirror backups and maybe over the next few days it will purge enough old stuff that it will start working cleanly.

Should I delete the entire remote target of the Mirror to get it started cleanly? Or, more likely, delete all backups on that Mirror for the 5 VMs that failed with this error.

@JamfoFL Any luck getting this to work? I've put off installing the patches because I don't want to lose my XO installs.

@olivierlambert I totally forgot about mirror backups, thanks for the reminder.

I see that a mirror will let me have different backup retention on the mirror target than I do on the mirror source. That's fantastic in my case because the mirror source has less storage space than the mirror target. Using rsync would have limited the backup retention on the Synology.

Now can you tell me how to recover the hours I spent figuring out I couldn't get it working to pull from the Synology side then the time to figure out how to schedule tasks in UNRAID and learn rsync and confirm it would do what I want it to do?

I have two remotes that I use for delta backups. One is on a 10Gbe network and the other only has a single 1Gbe port. Since XO writes those backups simultaneously, the backup takes a really long time and frequently randomly fails on random VMs.

I was thinking it would be better if I have XO perform the backup to the 10Gbe remote running UNRAID then schedule an rsync job to mirror that backup to the 1Gbe remote on a Synology array. I've already NFS mounted the Synology share to UNRAID and confirmed I can rsync to it from a prompt, I just need to create a cron job to do it daily. In addition to, hopefully, preventing the random failures, it would let XO complete the backup a lot faster and my UNRAID server and Synology will take care of the rest, reducing the load on my main servers during the workday. The current backup takes long enough that it sometimes spills into the workday.

I'm wondering if that will cause any restore problems. When XO writes to two remotes, does it write exactly the same files to both remote such that it can restore from the secondary even if it never wrote the files to that remote? I would assume they are all the same but want to make sure I don't create a set of backup files that won't be usable. Granted, I can, and will, test a restore from that other location but I don't want to destroy all the existing backups on that remote if this would be a known-bad setup.

Note, I did a --dry-run with rsync and see quite a few files that it would delete and a bunch it would copy. That surprises me because that is still an active remote for the delta backups. I'm suspicious that the differences are due to previously failed backups.

This happens to me on 8.2 with one specific VM. It doesn't always happen but about once a week (it feels like) the backup will get stuck in Starting and it's always, or almost always, the same VM it's stuck on. I usually reboot the VM running XO and that makes the backup task change to Interrupted and then the next scheduled backup usually works.

Would be nice to know if there's something that's causing this that I can try to fix. Not a world-ending problem but it's a nuisance since it stops all future backups for that Job until it's resolved. I'm happy that all the other VMs in that job get backed up at least.

@flakpyro Oh thank you. I try to go through every update post but I must have missed that one.

It worked

Makes a big difference for this VM if I back up a partially filled 250GB disk vs a largely full 8,250GB set of disks when 8TB of that doesn't need to be backed up.

Will this add the ability to control which disks on the VM will be backed up? I'd love to be able to select specific disks on a VM to backup and leave others out.

Maybe configured at the VM level rather than the backup level. Flag a disk as not needing backup and then the regular backup procedure would ignore it. However, I could also see why it might be better to control it by creating a specific backup for that VM so you could have different backup schedules, some that backup those extra disks and some that don't. I have no need to ever backup the extra disks at the moment though.

Looking through the kern.log files I found stuff I thought might be interesting but as I scroll, I see it happening a lot so I then wonder if it's normal. I see sets of three events:
Out of memory: Kill process ###
Killed process ###
oom_reaper: reaped process ###

I wonder if it was starving for memory and having to kill off processes to survive then eventually died. I see these 90+ times in one log file and over 200 times in another. Don't know if this is just normal activity or indication of a problem.

@olivierlambert Having trouble finding the reboot in the log files because I don't know what to look for. I have nearly 1GB of log files from that day and unfortunately, I don't recall when the reboot happened. Is there something I can grep for in the log that would indicate the reboot and I can backtrack from there?

Tried feeding the GB of log files to Phi4 in my Ollama server but so far it has not been any help finding anything either. Well, it found how to make my office a lot louder by running the server fans at full speed for a few minutes but that wasn't helpful.