@ph7 I'm seeing the same thing as you, where I'm getting a mismatch between the server that is sending out the alert and then ending the alert. Just like you, it is actually the XO server that is truly the one that should be alerting. The second server (and it's always the same second server) is NOT having any issues with CPU or memory usage but is being drug into the alerts for some strange reason.

I'm currently on Commit 2e8d3 running Xen from sources. Yes, I know I'm 5 commits behind right now, and will update as soon as I finish this message. However, this issue has been going on for me for some time now and when I saw others with the same issue, I figured I'd add to the chain.

One other thing that happed around the same time this issue started... it seems the Average Length value for alerts are being ignored, or are at least being handled differently than they had previously. For example, I have my CPU alert set to trip if it exceeds 90% for over 600 seconds. Before the issue started, if I had a long running backup, my CPU would go over 90% and could sometimes stay there for an hour or more. During that period, I would get a single alert after the CPU was over 90% for that period of time and then Xen was "smart" enough that it would keep an eye on the average, so a brief couple second dip below 90% would NOT send out an "end of alert" and then a second "alert" message when the CPU went over 90% once again. This is not happening anymore... if the CPU spikes over my 90% threshold, I get an almost immediate alert message. The instant the CPU goes below the 90% threshold, I get an immediate end of alert message. If threshold goes back over 90%, even a few seconds later, I get yet another alert message.

This has had the effect where instead of getting a single message that spans the duration of the time the threshold is exceeded, where brief dips below were ignored if they were only a few seconds long, I am now getting an alert/end of alert/alert sequence for every seconds-long dip in CPU usage. Last night, for example, I received over 360 alert e-mails because of this, with many happening within seconds:

So... just confirming what @ph7 has been seeing... alerts are sending out from one server and a second sends the end-of-alerts, and for some reason the ability of Xen to average the alerts over the selected period of time so messages aren't sent out with every single seconds-long dip below the threshold is no longer working, as well.

Thanks!

@julien-f I just ran a "yard build" this morning, and other than still seeing the chunk error message:

(!) Some chunks are larger than 500 KiB after minification. Consider:
- Using dynamic import() to code-split the application
- Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/guide/en/#outputmanualchunks
- Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.

Everything else ran fine... no errors or OOM issues.

@alcaraz73 @screame1 @AtaxyaNetwork @florent @Danp @olivierlambert I know I'm late back to the party, but just got back into our office after Hurricane Ian. Thanks to all for your well wishes... I was very fortunate to come out with no damage, and was only without power for about 12 hours.

Now that I'm back, I also applied the fixes here (and updated to the latest commit) and can confirm continuous replication is working like a champ.

Thanks to everyone involved for the help!

@Danp Current commit is 3d3b6.

xo-server: 5.103.0
xo-web: 5.104.0

Yes... as I had mentioned, everything worked until I updated yesterday at around 3:50 PM. I noticed there were five different commits that were released yesterday, and the one you linked to is one of those. So, as I figured, one of the commits from yesterday "broke" the ability to run continuous replications.

Now that it looks like you've focused on a cause, I'm sure it's only a matter of time until a new commit is published to fix the issue. Once it's released, I'll get it installed and, hopefully, we can put this behind us.

Thanks!!

@olivierlambert I would agree... one of those odd glitches that occurred during the build process that corrected itself on a second run.

Thanks for everyone's help... please go ahead and close this out!

I just wanted to chime in that I was having the same issue after updating to the latest commit earlier today, 1b5157e9a7a7ba9a49ebc9484737c34ef3da95ed.

I rolled back to my previous commit (granted, it's a bit of an older one as I hadn't updated since last week) and the CR backups started working again perfectly.

In my case, both XCP-ng hosts are on the same subnet.

So, something seems to have broken CR backups between last week and today. Fortunately, rolling-back got everything working again.

If there's anything I can do on my end that might help, just let me know!

@olivierlambert A-ha! Thanks! So it was just a coincidence... and I was a little ahead of the game.

Sorry for the confusion. Thanks for putting my head on straight!

@Melissa-FR Thank you, @Melissa-FR! I really appreciate the update.

@julien-f I just wanted to update and let you know that whatever changes you made fixed the issue for me permanently! I have been able to routinely update since that time and have had no further issues.

Thanks for all of your hard work, Julien-f!

OK... just an update. Everything has been working perfectly on bf51b94 for the past few days, so I have just updated to the next commit after that one, 263c23a.

I will continue to monitor and if things keep working, will move to the next commit in line.

@MathieuRA Thanks so much! I appreciate all the effort!

@MathieuRA Yes, I can confirm I am using the All Running VMs and All Running Hosts (I am not using All Running SRs, but I never get alerts for those because I have a LOT of free disk space).

I did place an exclusion for one of my VMs (the one that was generating dozens and dozens of alerts) to cut down on some of that chatter, but even with one machine excluded, when I do get a report from one of the other VMs it still has the same issue: the proper VM will generate the alert, but an improper VM will be reported in the end of alert message.

So... as far as I can tell, we still have the issue with the improper machine identification and the Average Length field is ignored so a machine that pops over the threshold, then briefly under the threshold for a few seconds, then back over the threshold again will generate three messages (alert, end of alert, alert) in several seconds instead of looking at the average to make sure the dip isn't just a brief one.

Hopefully that makes sense.

Thanks again!

@frank-s Think of it like this...

You take the original backup with snapshot on October 1st.

You then have a differential backup on October 8th. That differential only writes the changes that make the state of the server on October 8th different (hence the term differential) from the state of the server as it was on the full snapshot on October 1st.

Now you delete the snapshot of the full system state taken on October 1st, leaving only the October 8th differential.

It's now October 15th and you want to restore something from the October 8th differential and you figure you'll just make a new full snapshot from October 15th and then try to apply the October 8th differential. However, this won't work because the differential from October 8th will be missing key reference points it will be looking for in order to overlay its changes onto the snapshot. What else on that server changed from October 1st to October 15th that isn't contained in the differential? What changes occurred from October 8th to October 15th that are newer than what is on the differential? You could, potentially, brick your entire system if the differential started overwriting changes it sees from its system state that are actually newer than the data it contains.

Differentials are entirely dependent on the full snapshot on which they are based. Any new snapshot taken after their creation will be totally foreign to the differential... those differentials will be looking for very specific system states that existed at the time the original snapshot was created, and those will be completely different from the system state as it will appear on a new snapshot created after the differential.

@ph7 I'm seeing the same thing as you, where I'm getting a mismatch between the server that is sending out the alert and then ending the alert. Just like you, it is actually the XO server that is truly the one that should be alerting. The second server (and it's always the same second server) is NOT having any issues with CPU or memory usage but is being drug into the alerts for some strange reason.

I'm currently on Commit 2e8d3 running Xen from sources. Yes, I know I'm 5 commits behind right now, and will update as soon as I finish this message. However, this issue has been going on for me for some time now and when I saw others with the same issue, I figured I'd add to the chain.

One other thing that happed around the same time this issue started... it seems the Average Length value for alerts are being ignored, or are at least being handled differently than they had previously. For example, I have my CPU alert set to trip if it exceeds 90% for over 600 seconds. Before the issue started, if I had a long running backup, my CPU would go over 90% and could sometimes stay there for an hour or more. During that period, I would get a single alert after the CPU was over 90% for that period of time and then Xen was "smart" enough that it would keep an eye on the average, so a brief couple second dip below 90% would NOT send out an "end of alert" and then a second "alert" message when the CPU went over 90% once again. This is not happening anymore... if the CPU spikes over my 90% threshold, I get an almost immediate alert message. The instant the CPU goes below the 90% threshold, I get an immediate end of alert message. If threshold goes back over 90%, even a few seconds later, I get yet another alert message.

This has had the effect where instead of getting a single message that spans the duration of the time the threshold is exceeded, where brief dips below were ignored if they were only a few seconds long, I am now getting an alert/end of alert/alert sequence for every seconds-long dip in CPU usage. Last night, for example, I received over 360 alert e-mails because of this, with many happening within seconds:

So... just confirming what @ph7 has been seeing... alerts are sending out from one server and a second sends the end-of-alerts, and for some reason the ability of Xen to average the alerts over the selected period of time so messages aren't sent out with every single seconds-long dip below the threshold is no longer working, as well.

Thanks!

@olivierlambert I would agree... one of those odd glitches that occurred during the build process that corrected itself on a second run.

Thanks for everyone's help... please go ahead and close this out!

@olivierlambert I went through all of the tabs and everything is working as it should be. The only thing that was affected was that log on screen and all looks good now!

Is there something in particular you would like me to check for you?

@olivierlambert I always run the following procedure when applying Commits:

git checkout .
git pull --ff-only
yarn
yarn build

Never had any issues in the past. So I figured maybe something didn't complete quite right with the last run, so I re-ran the process again and it seems to have fixed the issue.

I guess it was just one of those weird things that can happen from time to time.

Thanks for getting back to me!

@probain Cleared, but did not help. Same result...

Exceedingly minor error... I'm really only posting just to let you know it is happening as the issue is purely cosmetic.

Xen Orchestra from sources. I just upgraded to Commit DE934. After applying the Commit, everything is working just fine but, for some reason, the log on screen is now missing its graphics and has a "messed up" appearance:

Oddly enough, this seems as if it is only happening in Google Chrome. Oddly enough, it seems to look fine on both Edge and Firefox.

Again, this isn't affecting anything else other than the appearance of the log on screen so the issue is purely cosmetic. I just wanted you all to be aware.

Thanks!

@julien-f I can confirm what @Andrew posted... I also updated to Master Commit 583c7 and the CR backups are working properly once again!

Thank you so much!!