Posts made by planedrop

@olivierlambert @florent Just following up on this again, I am going to be doing some more testing this week to see if I can discover any issues. Still very curious about how this should be handled though.

Been doing some testing recently and hoping to get some feedback or a better understanding of how this works.

I am using Backblaze (S3 compatible) for some backup testing, I have the bucket setup with immutable storage (Object Lock as they call it) with a 30 day retention period for testing.

I created a backup job to test with this remote with the below settings:

• 2 VMs set to backup
• Retention is 30
• Full Backup Interval is 15

What I am confused about is, what happens when a merge needs to happen? If it's a locked object, you can't merge data into the full backup.

So once the 30 backup retention period is hit, how does the delta backup merge blocks into the full backup of the chain? That should be impossible if the object is write restricted.

However, I am not getting any errors when running this backup job beyond the 30 backup retention setting. Shouldn't it error out since it can't write to the full backup VHD?

Or there is maybe something I'm not understanding here?

I'm also wondering how this is managed long term, since the objects can't be deleted, XOA will (I presume) try to delete them, that will fail, and then they are just there in the bucket forever since XOA isn't aware of the immutable retention period to go cleanup later.

@FTSSupport Yeah that's really annoying, oof, sorry to hear that.

Maybe the only option is going to be getting a new physical host and then using Hyper-V on that? Hate to say it, but sounds like it could be the final result.

I did that with ESXi for this one vendor since I didn't have a choice at the time, it was lame, but the company understood the need for the expense, though it helped that we were going to need another (albeit not as powerful) host anyway.

@arc1 Think you can give it a try with a Windows VM just to see if the problem goes away (not SQL but just pinging)? Would help diagnose if it's your infrastracture somehow or an XCP-ng specific thing with just certain Linux VMs.

I so far haven't seen behavior like this though.

@FTSSupport That's an interesting way to do it instead of just having an OVA file lol.

I'm honestly a little surprised any vendors require Hyper-V too, like, if you're going to require something, why not use the industry standard that is ESXi?

And if that was the case, it would be an OVA which would be something you could natively import to most hypervisors anyway.

What an interesting situation lol.

@FTSSupport Got it! I'm not sure there is a way around hyper-v complaining on that front unfortunately.

@FTSSupport Should be easier to move back to 8.2 than Proxmox, yes. I'm still not sure it's going to work that well though, either in 8.2 or in Proxmox. I've just never had a good experience with nested virt.

I got it working well on a Hyper-V setup once, but the nested VM still had some odd issues, bad latency, and a few other things, and that was all Windows based stuff so it was kinda a best case scenario.

Good luck with the vendor, hopefully they can be convinced that it will work just fine on other hypervisors.

One vendor I worked with that required ESXi for their Windows VM finally changed their minds and worked w/ me to do some validation on XCP-ng. After we ran through a ton of testing (this was a very high bandwidth/data usage platform with strict requirements) the engineers were flabbergasted because XCP-ng performed so much better lol. I was like "you're really surprised something is faster than bloated ESXi??"

@stormi Excited to see more progress on this for sure.

Still should never be done in a production setup though so I don't think there should be any rush ha!

Would be very cool to have in lab environments though.

@brezlord It's not in the sources version yet, at least as of a build I did a few days ago, haven't checked the actual code.

But if you want to test it out, you can do it with XOA free edition, not sure if it works without registering (but I think it does), but since it's free to do so there is no cost to try it out.

Do you know of any changes that lead up to this? It sounds like it could be some kind of hardware issue that cascaded into a larger problem, but that's a bit of a shot in the dark.

The VMs that rebooted initially, did you trigger the reboot, or did they crash and then auto-boot back up?

I've managed a solid number of XCP-ng installs and can't say I've ever seen anything quite like this.

I did have a host completely freeze up once and it would not respond to VM boot requests or anything else, but the VMs on it were still running, a reboot resolved that. This was a single host pool though so no idea how the rest of the pool would have behaved if it was the pool master.

This was on consumer grade hardware though so I strongly believe it was an issue w/ the hardware (possibly an error in RAM and I don't have ECC on this box).

So right now, you basically can't boot or manage anything in your pool? If anything seems to work right, please define what is working right.

Also, if you have pro support, I'd recommend putting a ticket in with Vates, they're support is actually quite great.

As an update here, rebooting did resolve the issue, I thought I did so shortly after I did the restore but maybe I forgot to do so.

Anyway, rebooted yesterday and the backups ran last night just as they should.

@Danp Oh yes, I have also tried that, no luck though.

I think I tried re-saving the schedule, backup job, and disabling/renabling it all at the same time, but none of those made a change.

It may be worth noting I went from a Ubuntu install to a Debian install for XO from sources, but I don't think that would have anything to do w/ this.

I also did build this one via the github script a lot of people use that installs and updates XO from the sources. Vs my previous install was all built manually.

Sure but the solution to that is usually to buy a single server and then try each hypervisor on it in a lab type setup to see which you prefer the best.

I still agree that it would be nice to have 1 box that runs them all, for sure, but there aren't any that can actually do that.

@FTSSupport OH gotcha, so the software needs Hyper-V, and Hyper-V was complaining about the lack of ECC?

@FTSSupport Gotcha, yeah I think the first step would be finding out if there are workarounds from the vendor, then go from there if there aren't. May just require spinning up a physical Hyper-V box, but that would definitely suck. The entire idea of hypervisors was to not have to run so many different bare metal installs of things lol.

Good luck!

I may play around with nested Hyper-V here soon to see what I can get working, but honestly I'd still avoid it EVEN IF we can get it working stable, could be a headache to troubleshoot down the road.

@Danp I believe so, but I will reboot the entire VM today and see if they resume tonight just in case.

@ph7 yes, I have both tried opening and just saving the backups themselves, and opening the backups, opening the schedule, saving the schedule, then saving the backups.

After I try a reboot, my next step is going to be to delete and re-add the schedules.

@FTSSupport I'd first consider talking to the client and/or the clients vendor that is requiring this. Nested virtualization, even on setups that work, is NEVER recommended for something that requires good up time and reliability, so it might be best to avoid this.

I have had issues with nested virt in ProxMox as well, admittedly less than in XCP-ng, but it's still not good enough on either that I'd use it consistently.

Is there a reason the application needs Hyper-V? The reason I ask is, I've had something similar due to a vendor who required ESXi for their VMs. Turns out there was no good reason for that and we ended up having to spend a ton of money for no real benefit. Has created more headaches down the road too w/ lack of central backups, etc... and massive wasted resources (time, money, all of the above). The vendor has since said they'll help us move to XCP-ng, but it's a big process and there was no reason for ESXi in the first place.

So maybe, just maybe, there's a chance the vendor/application developer can be convinced otherwise? One of the points of virtualization is keeping options open.

There isn't a need for the VMs to be aware of it being ECC memory, that's something for the host to deal with, so it won't make a difference in Windows.

What is your purpose/requirement for Hyper-V? Nested virtualization is almost always unreliable, this isn't a XCP-ng or Xen thing, it's also buggy/unstable in ProxMox, other KVM hypervisors, etc... The only time I've had this work is nesting Hyper-V inside a VM that is already running on Hyper-V, but even that is something Microsoft says don't do.

I recently had to do a restore of my XO config to a new VM, this is built from the sources since it's in my lab. The config restore worked good and all things appear correct, however my backups have stopped running.

There are no errors or failures at all, and when I run the backups manually they work fine, so it seems the schedule isn't being triggered to star them.

I have tried going into the backups and then saving them again, but that hasn't changed anything.

Anyone seen this before?

I will continue to try a few more things to see if I can get the schedules to come back up. IMO they should "just work" though when restoring from a config file.

@olivierlambert issue added, it's my first time doing it on GH so apologies if I missed anything or put it in the wrong category.

https://github.com/vatesfr/xen-orchestra/issues/7893

planedrop created this issue in vatesfr/xen-orchestra

open Add Email Notification For License Issues To Prevent Backup Failure #7893