CodeMercenary

I got this running and when I set up a catalog and baseline the report tells me that it has nothing in the catalog. Is that because it doesn't understand XCP-ng or because my hardware is old enough that it has dropped out of the catalog? I have two PE R630 and one PE R730. I tried both the full default Dell catalog as well as the one that limits to BIOS and firmware updates.

@poddingue These are VMs that have been doing full backups fine for a very long time. I have run into the too-much-free-space issue in the past with a different brand new VM but these have been around since I migrated them from VMware during the Broadcom fiasco years ago. It is also totally inconsistent which VMs fail with the body timeout. One day it'll be a single one, the next day it'll be three or four and won't include that one, then it'll be a couple other ones unrelated to any of the earlier ones, then I'll have no failures. It's all over the place. I'm just glad that over the course of several days I get good backups of everything.

Oh, a maybe useful data point is that my delta backups that all succeed are handled by a different instance of XO that runs on a different one of the hosts. I might be overly paranoid, but I run full backups and delta backups of the same VMs using different XO instances to different target remotes, at different non-overlapping times. This is part of the reason that I'm not hair-on-fire worried about the failing full backups.

A while back I stupidly let one of my Storage Repositories run out of space and it was the one hosting my main XO instance that handled all the backups. It took a while to get it back going again and I realized I wanted redundancy in what handled backups and moved the deltas over to another XO instance. Too many eggs in that one basket. I also wasn't able to restore the backup of that XO instance because that XO instance is what handled the backups. Thankfully it all worked out in the end, plus a number of extra gray hairs it gave me.

I checked my hosts and the clocks all seem to be very close in time, certainly within a second of each other. The one difference is that the host that is often the target for the backups is set to UTC and the other two are set to MST. Clearly an oversight on my part, unless it is purely a display issue. When I ssh into dom0 and use date, those two show MST and the other shows UTC. If that has ever changed, it would most likely have been in the upgrade from 8.2 to 8.3, which I performed on Jan 28.

All that said, this body timeout issue happens to me every day on random VMs within my backup. Occasionally a backup will complete with none of these errors on any VM but that's uncommon. All of the failed backups are from VMs or configs on the two MST hosts and are stored on the UTC host. (To be clear, they are also stored elsewhere.) I also have an XO instance on one of the MST hosts that performs delta backups and those have never failed. I don't know if that's because they are delta or if it's because they don't touch the UTC host.

The recent failed backups started on 6/26 for my VM full backups and on 6/29 for the config backups. Prior to that the config backups had never had a problem and the full backups had been fine back into February. From February 21 to March 12, I had a smattering of backup failures due to body timeout error for the full backups.

I see this Body Timeout Error frequently, almost every day, but for random VMs. It has been happening for weeks but since it's not consistently the same VM I figured overall I still have good backups. I also back up the VMs in different ways and the others aren't failing. It has more recently been happening with the config backups which should be very short. Previously I wondered if it could be increased network activity that messes with a long backup of a big VM but that would not be the case for config backups.

The failing backups are all network local, just to be clear, I am not streaming this backup to Backblaze or something where internet connectivity might be causing it.

I got my R730 working, thanks to @majorp93 giving the tip about dmidecode. Thanks a bunch. It turns out that while this is a Dell R730, it was actually sold by a different vendor for use as video storage so the internal product name doesn't have Dell in it, it's HD NVR. I used that as the vendor field and tweaked a couple other plugin settings and now it gives me data.

I am still missing two of the values in the UI and I'm not sure why, but for different reasons.

On my R630 servers they show the server IP address under the IPMI badge. They also both have the default Dell /^ip address$/i as the regex in the ip setting in the plugin. I used that same regex in the R730 and I can see in the ipmitool listing that it has the same field name for the IP address as the R630, yet it shows IPMI: Unknown.

Also confusing is that the R630 servers have nothing under fanStatus in the plugin settings but they show 14x fans status: ✅ in the UI. The R730 also has nothing under the plugin settings fanStatus field but it has no data in the UI. Looking at the ipmitool output I am not sure what field it would use for fan status. It makes me think there must be a default field name it is trying with the R630 to get that status but that doesn't work with the R730.

Any hints about how to get those last two badges to work on the R730?

I am also not sure what bmcStatus should be set to. I don't know what that means exactly so I don't know if any of the fields shown in ipmitool are applicable.

This plugin is really cool, thank you so much for building it.

@TheNorthernLight Thanks for pointing that out, that totally makes sense, I should have realized to look it up that way. I guess I just think of XCP-ng as its own product so much that I forget it's based on Xen even though I know that.

If anyone can point me to documentation on this, I would greatly appreciate it. I have looked in the docs and searched for an answer and come up empty.

I have two PE R630 and one PE R730xd. The two R630s show the IPMI data in XO but the R730xd does not. If I run ipmitool sensor from an ssh session, I get very similar data back on the 730 and 630.

This isn't critical to me, just letting you know that there are cases that functioning ipmitool on a Dell does not seem to be enough to show it.

There are some differences in the labels. The R730 has "Fan1 RPM", "Fan2 RPM" while the R630 has "Fan1A", "Fan2A", "Fan1B". Most of the other interesting fields seem to have the same names but the order is a bit different and I didn't do an exhaustive comparison.

It's very cool that you are surfacing this data now.

I was rebooting my servers due to the patches released last week and I notice that the BIOS in my server was not set to interleave the memory. Since I have symmetric memory sticks, it seems like it should be set to interleave. However, the BIOS does say that for NUMA aware OSs, that interleaving should be left to the OS. I don't recall now, and my notes don't say, if I intentionally left interleaving off because v8.2 I was installing at the time was NUMA aware.

Memory interleaving on a Dell R630: on or off?

During the last patch I turned on interleaving for my R730xd (without reading the help that says not to do it with NUMA aware OSs) and it feels faster to me, but I didn't do any real testing so that could be wishful thinking bias. It certainly doesn't seem to have hurt performance.

@nikade I did install the 97 XCP patches and that night the VM that was failing to backup was able to successfully backup. It had a 500GB secondary drive that was formatted but totally empty.

I had not yet patched my XO from source to the latest, it was maybe 24 commits behind. On Friday I updated it so I will find out on Monday if those backups continued to work but I assume they did.