@jr-m4 I just wanted to report that today, February 24, 2025, that I updated to Commit f18b0 and that seems to have fixed the issue. I am able to see all drop downs once again and everything seems to be working normally.

@Danp and Support Team... thank you for looking into the issue!

@ph7 I'm seeing the same thing as you, where I'm getting a mismatch between the server that is sending out the alert and then ending the alert. Just like you, it is actually the XO server that is truly the one that should be alerting. The second server (and it's always the same second server) is NOT having any issues with CPU or memory usage but is being drug into the alerts for some strange reason.

I'm currently on Commit 2e8d3 running Xen from sources. Yes, I know I'm 5 commits behind right now, and will update as soon as I finish this message. However, this issue has been going on for me for some time now and when I saw others with the same issue, I figured I'd add to the chain.

One other thing that happed around the same time this issue started... it seems the Average Length value for alerts are being ignored, or are at least being handled differently than they had previously. For example, I have my CPU alert set to trip if it exceeds 90% for over 600 seconds. Before the issue started, if I had a long running backup, my CPU would go over 90% and could sometimes stay there for an hour or more. During that period, I would get a single alert after the CPU was over 90% for that period of time and then Xen was "smart" enough that it would keep an eye on the average, so a brief couple second dip below 90% would NOT send out an "end of alert" and then a second "alert" message when the CPU went over 90% once again. This is not happening anymore... if the CPU spikes over my 90% threshold, I get an almost immediate alert message. The instant the CPU goes below the 90% threshold, I get an immediate end of alert message. If threshold goes back over 90%, even a few seconds later, I get yet another alert message.

This has had the effect where instead of getting a single message that spans the duration of the time the threshold is exceeded, where brief dips below were ignored if they were only a few seconds long, I am now getting an alert/end of alert/alert sequence for every seconds-long dip in CPU usage. Last night, for example, I received over 360 alert e-mails because of this, with many happening within seconds:

So... just confirming what @ph7 has been seeing... alerts are sending out from one server and a second sends the end-of-alerts, and for some reason the ability of Xen to average the alerts over the selected period of time so messages aren't sent out with every single seconds-long dip below the threshold is no longer working, as well.

Thanks!

@julien-f I just ran a "yard build" this morning, and other than still seeing the chunk error message:

(!) Some chunks are larger than 500 KiB after minification. Consider:
- Using dynamic import() to code-split the application
- Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/guide/en/#outputmanualchunks
- Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.

Everything else ran fine... no errors or OOM issues.

@alcaraz73 @screame1 @AtaxyaNetwork @florent @Danp @olivierlambert I know I'm late back to the party, but just got back into our office after Hurricane Ian. Thanks to all for your well wishes... I was very fortunate to come out with no damage, and was only without power for about 12 hours.

Now that I'm back, I also applied the fixes here (and updated to the latest commit) and can confirm continuous replication is working like a champ.

Thanks to everyone involved for the help!

@Danp Current commit is 3d3b6.

xo-server: 5.103.0
xo-web: 5.104.0

Yes... as I had mentioned, everything worked until I updated yesterday at around 3:50 PM. I noticed there were five different commits that were released yesterday, and the one you linked to is one of those. So, as I figured, one of the commits from yesterday "broke" the ability to run continuous replications.

Now that it looks like you've focused on a cause, I'm sure it's only a matter of time until a new commit is published to fix the issue. Once it's released, I'll get it installed and, hopefully, we can put this behind us.

Thanks!!

I just wanted to post an update as of Monday, February 24, 2025...

I updated to Commit f18b0 and that seems to have fixed the issue. All of the dropdown options that were missing in the above screen shots, as well as many others, are all back and responding normally once again. So, it looks like the issue has been addressed and operations have returned to normal.

The only oddity I now see after all of that is when I run yarn after pulling the updates, I receive an error message that the expected version of Node is now v20. I upgraded to v20, but it seems that the backup portion of Xen still needs v18... so be sure to keep both on the system or things will break. I have both v18 and v20 installed and running, with v20 set as default.

Other than that little oddity, it seems everything else is doing everything it should be doing!

Thanks to the excellent Vates teams for getting everything back on track.

@probain Thanks for the update... I will monitor that ticket, as well.

@olivierlambert I would agree... one of those odd glitches that occurred during the build process that corrected itself on a second run.

Thanks for everyone's help... please go ahead and close this out!

I just wanted to chime in that I was having the same issue after updating to the latest commit earlier today, 1b5157e9a7a7ba9a49ebc9484737c34ef3da95ed.

I rolled back to my previous commit (granted, it's a bit of an older one as I hadn't updated since last week) and the CR backups started working again perfectly.

In my case, both XCP-ng hosts are on the same subnet.

So, something seems to have broken CR backups between last week and today. Fortunately, rolling-back got everything working again.

If there's anything I can do on my end that might help, just let me know!

@olivierlambert A-ha! Thanks! So it was just a coincidence... and I was a little ahead of the game.

Sorry for the confusion. Thanks for putting my head on straight!

@Melissa-FR Thank you, @Melissa-FR! I really appreciate the update.

@julien-f I just wanted to update and let you know that whatever changes you made fixed the issue for me permanently! I have been able to routinely update since that time and have had no further issues.

Thanks for all of your hard work, Julien-f!

OK... just an update. Everything has been working perfectly on bf51b94 for the past few days, so I have just updated to the next commit after that one, 263c23a.

I will continue to monitor and if things keep working, will move to the next commit in line.

@olivierlambert OK... I am happy to help!

Let me take a little time to capture good, working backups of my lab environment so I have that peace of mind, and then I will start applying each of the subsequent commits until I can report the specific commit that starts the issue.

This may take a little time, as I can report when it first happened (on Monday, Feburary 6th) everything did seem to work for several hours after I applied the three commits that were published that day:

• 2f65a86aa08a8c05de0f5d864994b560f528d364
• 2a70ebf66711030cf6e277aeabc64037548b9a6b
• 55920a58a32e01a3ea0b966c8f6f542f835e936a

It wasn't until well into the evening of Monday night when the first CR backups started to fail, and everything seemed to cascade from there. So, I will incrementally apply each commit and will let it run for a day or two (unless if fails sooner, of course). Once I have the definitive commit where things break, I will report that back right away.

Thanks you for all your help!

@julien-f Thank you, Julien-f!

I ran through that procedure (sorry it took so long) and the last commit where everything works as expected was bf51b94. Of course, I could be back to the point where it might need to run all day before one fails, but I was, at least, able to run one of each type of backup on that commit.

Is there any way I can revert my environment back to that commit so I can have the security of backups again? Or should I ride this out while you take a look?

Thanks!

Jumping in as another "victim"... I have lost almost all ability to run any kinds of backups.

System information:

XO from Sources
XO-Server: 5.109.0
XO-Web: 5.111.0
Commit: d4bbf

I had no issues running any kinds of backups prior to applying the latest commits two weeks ago, Monday, February 6, 2023. After applying that commit, everything seemed to be OK until late that evening. At that time, one of my CR jobs started and never completed. All of the subsequent jobs failed, of course, because another job was already running. I was able to delete the job and restarted the toolstacks and rebuilt a new job. This started to spread until, eventually, all of my CR jobs would fail. That issue has now spread and my DR jobs are now failing, too. Out of the seven backup jobs that were working with no issues prior to that date, I am now only able to run one job successfully, and that is a DR job.

The DR and CR jobs are written to different repositories (the CR jobs are written to the storage repository on one of the other XCP-ng servers and the DR jobs are written to an NFS share).

I tried running the cr-issue branch update that @julien-f recommended, and modified my config.toml file with the ignorePrematureClose = true code that was also recommended. Nothing works.

If I try to run a DR job now, even a brand-new one, it looks like it wants to start and create the snapshot, but then fails and actually shows it will start in 53.086 years.

Anything I can get, just let me know what to pull and I'll add it here... hopefully that will help. Right now, I'm able to only back up a single VM!

@MichaelCropper That's awesome! I'm glad the usual computer pixie dust fairies dropped by and fixed the issue for you.

Poor Windows though... you initially laid the blame at their feet, but it seems more like this was a Google Chrome issue. I've noted a LOT of peculiarities with Google Chrome as of late and have started to migrate away for that reason. Reading a lot of forums on Reddit and other IT sources, this seems to be something that is pretty wide-spread.

In any case, it's working for you now... until Google changes something again. LOL

I hope you have a great weekend!

@Danp I'm using Debian Bullseye... I am able to run the updates now, but still see that same "chunk size" error notice you initially reported. Still, it seems you have it even worse...

@MichaelCropper Can you tell me what dark mode theme you installed? I was attempting to test for you to see if I could confirm if I was seeing the same as you when using Chrome in dark mode.

However, even after switching Windows to dark mode, and then downloading a "dark mode" theme for Chrome, as soon as I open Orchestra, the screen is the normal bright white screen from my screenshot (and what you see in Edge). If you can let me know how you set your Chrome browser into your current dark mode theme, I will try and replicate. If it turns out I have the same issue, that might help @olivierlambert down the line with figuring out why that specific configuration causes the console window to disappear.

Thanks, Michael!