CR - Cannot start copy because suspended
-
Two things, and the first is a correction to what I posted earlier.
I was wrong about the memory option. I said your reasoning matched the docs, and it does, but the docs are wrong on this point. I checked with the backup team: we do not handle application consistency in backups at the moment, quiescing was removed, and restoring a domain controller from a RAM image can roll replicated state backwards and cause replication problems. Nobody would vouch for SQL Server either way. So "with memory" is not the safer choice for AD or SQL, and it may be the riskier one. That was my mistake, and you acted on it.
On this morning's failure, the error is worth reading closely.
VM_BAD_POWER_STATE(..., halted, suspended)means XAPI wanted the VM halted and found it suspended. That is the same mismatch you hit at the top of this thread with "Start a copy", except it is now breaking the delta job that ran fine for months before you added the CR.Two things would help pin it down: does it fail the same way tonight, and does the delta job succeed if you disable the CR job for a single run? If it does, that is a clean reproduction and worth a GitHub issue with this screenshot.
@Team-XO-Backend, this one now has an XAPI error and a clear before and after, so it has moved past being a docs question.
-
P poddingue marked this topic as a question
-
Thank you for that answer.
I will make the following changes:- Switch the snapshot to normal
- Turn off replicated VMs
- Lanch the job
Is there any documentation with details about the different snapshot modes (Normal, with memory, or offline)?
Thanks.
-
On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the
VM_BAD_POWER_STATEon a delta job that worked for months is the interesting part.If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.
Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.
-
Unfortunately, I need to have a CR job that works because we have a constraint this weekend.
However, I'm available to provide you with logs if necessary.I could let you know if changing the status of the replicated VM (from suspended to forced shutdown) has an impact on the backup chain. (with normal snapshot).
-
Getting your backups working for the weekend is the right call.
If you do flip the replica from suspended to a forced shutdown, whatever you notice about the chain afterwards would be worth a line here.
That is the one question I could not answer, and I have not found it written down anywhere either.
On the logs, I would wait until someone from the storage side actually asks, so you are not gathering things nobody reads. TheVM_BAD_POWER_STATEfrom post #3 is with @Team-Storage now, though I honestly do not know how fast that gets picked up.
Sorry again for the memory advice, and thanks for staying patient with a thread that changed direction twice on you. -
So, as planned, I changed the snapshot to “normal” in the backup job and performed a forced shutdown on the replicated VMs.
This morning, the job status is “successful,” but two VMs (both of which have the ADDS role—this may be a coincidence) are showing this warning:
The type is still delta despite the warning message.
For alle the other VM, the backup seems ok.
-
@Dezerd What does "xe task-list" show as far as active processes?
-
It's empty. Why ?
-
@Dezerd To check if there is perhaps a hung process that is interfering. I assume there is sufficient memory and storage space. Do any I/O disk errors show in /var/log/SMlog ? Just to clarify. are there VMs that do not show any such error condition?
-
You did the experiment I asked for and then some, thank you.
One caveat before anyone reads too much into it: you changed two things at once, the snapshot mode AND the replica power state, so the "fell back to a full" on the two ADDS VMs cannot be pinned on the forced shutdown by itself, right?
The line I would look at in your screenshot is "failed to open disk" under Clean VM directory, since that step works on the existing chain rather than on the new transfer, and it fits a fallback to full better than the snapshot mode does.
@tjkreidl'sSMlogquestion is the right next step and I would follow that rather than anything from me.
I still cannot tell you what the power-state change did to the chain.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login