@MathieuRA Thanks so much! I appreciate all the effort!

@MathieuRA Yes, I can confirm I am using the All Running VMs and All Running Hosts (I am not using All Running SRs, but I never get alerts for those because I have a LOT of free disk space).

I did place an exclusion for one of my VMs (the one that was generating dozens and dozens of alerts) to cut down on some of that chatter, but even with one machine excluded, when I do get a report from one of the other VMs it still has the same issue: the proper VM will generate the alert, but an improper VM will be reported in the end of alert message.

So... as far as I can tell, we still have the issue with the improper machine identification and the Average Length field is ignored so a machine that pops over the threshold, then briefly under the threshold for a few seconds, then back over the threshold again will generate three messages (alert, end of alert, alert) in several seconds instead of looking at the average to make sure the dip isn't just a brief one.

Hopefully that makes sense.

Thanks again!

@frank-s Think of it like this...

You take the original backup with snapshot on October 1st.

You then have a differential backup on October 8th. That differential only writes the changes that make the state of the server on October 8th different (hence the term differential) from the state of the server as it was on the full snapshot on October 1st.

Now you delete the snapshot of the full system state taken on October 1st, leaving only the October 8th differential.

It's now October 15th and you want to restore something from the October 8th differential and you figure you'll just make a new full snapshot from October 15th and then try to apply the October 8th differential. However, this won't work because the differential from October 8th will be missing key reference points it will be looking for in order to overlay its changes onto the snapshot. What else on that server changed from October 1st to October 15th that isn't contained in the differential? What changes occurred from October 8th to October 15th that are newer than what is on the differential? You could, potentially, brick your entire system if the differential started overwriting changes it sees from its system state that are actually newer than the data it contains.

Differentials are entirely dependent on the full snapshot on which they are based. Any new snapshot taken after their creation will be totally foreign to the differential... those differentials will be looking for very specific system states that existed at the time the original snapshot was created, and those will be completely different from the system state as it will appear on a new snapshot created after the differential.

@ph7 I'm seeing the same thing as you, where I'm getting a mismatch between the server that is sending out the alert and then ending the alert. Just like you, it is actually the XO server that is truly the one that should be alerting. The second server (and it's always the same second server) is NOT having any issues with CPU or memory usage but is being drug into the alerts for some strange reason.

I'm currently on Commit 2e8d3 running Xen from sources. Yes, I know I'm 5 commits behind right now, and will update as soon as I finish this message. However, this issue has been going on for me for some time now and when I saw others with the same issue, I figured I'd add to the chain.

One other thing that happed around the same time this issue started... it seems the Average Length value for alerts are being ignored, or are at least being handled differently than they had previously. For example, I have my CPU alert set to trip if it exceeds 90% for over 600 seconds. Before the issue started, if I had a long running backup, my CPU would go over 90% and could sometimes stay there for an hour or more. During that period, I would get a single alert after the CPU was over 90% for that period of time and then Xen was "smart" enough that it would keep an eye on the average, so a brief couple second dip below 90% would NOT send out an "end of alert" and then a second "alert" message when the CPU went over 90% once again. This is not happening anymore... if the CPU spikes over my 90% threshold, I get an almost immediate alert message. The instant the CPU goes below the 90% threshold, I get an immediate end of alert message. If threshold goes back over 90%, even a few seconds later, I get yet another alert message.

This has had the effect where instead of getting a single message that spans the duration of the time the threshold is exceeded, where brief dips below were ignored if they were only a few seconds long, I am now getting an alert/end of alert/alert sequence for every seconds-long dip in CPU usage. Last night, for example, I received over 360 alert e-mails because of this, with many happening within seconds:

So... just confirming what @ph7 has been seeing... alerts are sending out from one server and a second sends the end-of-alerts, and for some reason the ability of Xen to average the alerts over the selected period of time so messages aren't sent out with every single seconds-long dip below the threshold is no longer working, as well.

Thanks!

@olivierlambert I would agree... one of those odd glitches that occurred during the build process that corrected itself on a second run.

Thanks for everyone's help... please go ahead and close this out!

@olivierlambert I went through all of the tabs and everything is working as it should be. The only thing that was affected was that log on screen and all looks good now!

Is there something in particular you would like me to check for you?

@olivierlambert I always run the following procedure when applying Commits:

git checkout .
git pull --ff-only
yarn
yarn build

Never had any issues in the past. So I figured maybe something didn't complete quite right with the last run, so I re-ran the process again and it seems to have fixed the issue.

I guess it was just one of those weird things that can happen from time to time.

Thanks for getting back to me!

@probain Cleared, but did not help. Same result...

Exceedingly minor error... I'm really only posting just to let you know it is happening as the issue is purely cosmetic.

Xen Orchestra from sources. I just upgraded to Commit DE934. After applying the Commit, everything is working just fine but, for some reason, the log on screen is now missing its graphics and has a "messed up" appearance:

Oddly enough, this seems as if it is only happening in Google Chrome. Oddly enough, it seems to look fine on both Edge and Firefox.

Again, this isn't affecting anything else other than the appearance of the log on screen so the issue is purely cosmetic. I just wanted you all to be aware.

Thanks!

@julien-f I can confirm what @Andrew posted... I also updated to Master Commit 583c7 and the CR backups are working properly once again!

Thank you so much!!

I just wanted to chime in that I was having the same issue after updating to the latest commit earlier today, 1b5157e9a7a7ba9a49ebc9484737c34ef3da95ed.

I rolled back to my previous commit (granted, it's a bit of an older one as I hadn't updated since last week) and the CR backups started working again perfectly.

In my case, both XCP-ng hosts are on the same subnet.

So, something seems to have broken CR backups between last week and today. Fortunately, rolling-back got everything working again.

If there's anything I can do on my end that might help, just let me know!

@julien-f Agreed... I will add (just for future reference) that I did confirm that I was on the very latest versions of xo-server at the time... in fact, I made sure to apply all updates and reboot the Xen server before submitting the ticket.

Nothing seemed to work... even multiple reboots, etc. Once @Danp made mention of the Nodejs version and I updated from v18.18.2 to v18.19.1 everything immediately started working. I made no other changes after applying the Nodejs update.

So while that may not have been the cause, it somehow fixed whatever was ailing me. I have no explanation either, other than the fact that making that change is what took me from the feature not working to the feature working.

@julien-f Thanks for following up. I don't know if you saw the whole thread, but both master and Xen Orchestra were updated with the latest commits before I started and that did not seem to help. Once I updated Nodejs from 18.18.2 to 18.19.1 everything worked as expected.

I can't do any further testing as things were because everything is working now.

It could be that there was something slightly messed up in my Nodejs install and updating to 18.19.1 fixed that issue and, by extension, the problem I was having in Orchestra. My original error log message is posted above, if that helps.

@Danp Yup... that fixed it. Seems like such a minor revision update of Nodejs, but I guess it was the secret sauce.

Thanks for your mental prowess! I really appreciate your help.

Please feel free to close this out!

@Danp OK... I'll try updating and see if that makes a difference!

@Danp It reports, "v18.18.2".

@Danp Yes... it happens on every VM in the pool, no matter on which host they happen to reside.

Good morning and happy Leap Day to my fellow Xen Orchestra folks!

I'm having an issue where the new "VM Creator" drop down on the Advanced tab for my virtual machines does not appear to work. I can click the arrow to bring up the list of all the associated users, but once I select any of the users I receive an error log.

First, the Orchestra details. It is 9:15 AM EST on February 29th, and I just installed all of the latest commits. Both the Master and Xen Orchestra commits report #2560b.

As soon as I click on any of the names in the drop down, I receive an "Invalid parameter" error message. Here's the full log:

vm.set
{
  "creation": {
    "user": "06168cdb-3e03-464d-b623-015d8f5aeb79"
  },
  "id": "d886a884-82c6-d68b-419f-f26279eff06d"
}
{
  "code": 10,
  "data": {
    "errors": [
      {
        "instancePath": "",
        "schemaPath": "#/additionalProperties",
        "keyword": "additionalProperties",
        "params": {
          "additionalProperty": "creation"
        },
        "message": "must NOT have additional properties"
      }
    ]
  },
  "message": "invalid parameters",
  "name": "XoError",
  "stack": "XoError: invalid parameters
    at Module.invalidParameters (/root/xen-orchestra/packages/xo-common/api-errors.js:26:11)
    at Xo.call (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:92:22)
    at Api.#callApiMethod (file:///root/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:441:19)"
}

Hopefully that makes sense to the bigger brains out there.

Not a big deal, really... just an error that popped up!

@olivierlambert A-ha! Thanks! So it was just a coincidence... and I was a little ahead of the game.

Sorry for the confusion. Thanks for putting my head on straight!

@olivierlambert Maybe it's two different things?

After applying all the latest updates as of about 90 minutes ago, I now have this new "Edit VM Notes" button on my screen:

I had never seen that before... and I last updated on Wednesday of last week.

I thought that might be what @Melissa-FR was talking about, but maybe this is another new, unrelated feature?