random backups failing
-
Replying with an update. As usual, a manually trigger backup completes. 12 minutes to send to the remote, 10 minutes to bring it back. But with the domain controller, it look like I may need to update the management agent, it's taking a long time to get that running.
When a DC comes up, it reaches out to the network to check with it's peer and sync data while it was gone. During the health check the network is not there so it takes 5+ minutes to fully boot. it's now sitting waiting. took about 10 minutes total for the MA to report and then shut the health check down. Not sure if there is a better way to do this for a domain controller or not, mostly just a data point for other people that think their health check may be stuck.
Time to go fix the other VM that failed last night.
[edit] it said it failed again, though I watched the console for about 4 minutes and clearly it booted. I'll have to update the MA and drivers and see what happens. But it looks like 10 minutes waiting for the MA is too long now.
-
Probably too long ago, mentioned it on my follow up post while you were posting.
-
@Greg_E Worth remembering that DC backups need to be completed by something that is database aware, as far as I know, XO doesn't currently have this functionality, and if so, you are risking your entire environment....
-
@DustinB Hmmm... They have always completed in the past, but I'll admit that I haven't tried to restore one and see if it really works. too busy to perform the test the last couple years.
but now I know the time out is after the machine boots, and I've seen the console come up and wait longer than 4 minutes, so probably it is OK if it could connect to the other DC and compare the database.
-
@Greg_E Right, the backup operations are occurring outside of the VM, at the hypervisor layer. The guest drivers, don't stop or flush any database writes to disk etc.
So while the VM will likely boot, you might lose records or changes.
Depending on the size of your organization, it may be moot, but still worth keeping an eye on it.
I've seen other people use Veeam Backup and Replication to backup their Domain Controllers outside of the XO backup mechanism, and now that it's officially integrated, it might be a good contender here (and for others). https://www.veeam.com/products/free/backup-recovery.html for things that need to be handled "carefully".
-
Ok, generally I'm doing backups after everyone has gone home for the night, so changes to the DB should be minimal.
I'll have to look into this more deeply and maybe use the Server Backup tool to schedule both DC going forward. Used to use that when bare metal, and had to restore a couple times too.
-
I'm guessing this might be having updates applied, and now I need new Xen Tools. I have a feeling tomorrow will be filled with updating the tools and checking things because I bet all my backups fail tonight. And I bet all fail after they try to start.
-
I'm guessing this might be having updates applied, and now I need new Xen Tools. I have a feeling tomorrow will be filled with updating the tools and checking things because I bet all my backups fail tonight. And I bet all fail after they try to start.
Are you performing nightly healthy checks? Seems obsessive to need to check every system makes it to a login screen, but on the other hand, I understand.
Just curious.
-
@DustinB Everything I care about gets backed up and health checked each night, some things really should be several times a day, but 1 day is tollerable.
I'll have to get things updated after I deal with some php2shell stuff, that will need to come first. And now node.js in my XO machines.
-
P poddingue marked this topic as a question
-
Pretty sure I figured this out. I went to update the Xenserver tools, and the updater said the installer was busy. That meant the machine was in a "dirty condition" and couldn't work right. The dirty condition was the newest Xenserver tools needing a couple more reboots to get fully insatalled.
testing now and will have to wait on the DC until the network use get lighter this afternoon.
Running the second VM right now, but pretty sure it will now pass the health check. I'll update after I check the DC in a few hours.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login