CR - Cannot start copy because suspended

Dezerd

Hello,

Any answers to these questions?
This morning, the backup didn't run correctly (the first delta backup after yesterday's full backup). For your information, this is a delta job that has been running perfectly for months. I simply added the CR.

For your information, we have not changed the status of the CR VM; it has not changed.

Here are the errors

poddingue

On your second question I still can't give you a firm answer, and it has now been asked twice, so @Team-XO-Backend would be the people to confirm what changing the replica's power state does to the chain.

On whether "With memory" is the right call, your reasoning does match what the docs say. The "Consistent backup" section is marked deprecated and says RAM enabled backup replaces it: https://docs.xen-orchestra.com/xo5/backups#consistent-backup. So for AD or SQL Server, that is the route the docs point you at now rather than the old quiesced one.

poddingue

Two things, and the first is a correction to what I posted earlier.

I was wrong about the memory option. I said your reasoning matched the docs, and it does, but the docs are wrong on this point. I checked with the backup team: we do not handle application consistency in backups at the moment, quiescing was removed, and restoring a domain controller from a RAM image can roll replicated state backwards and cause replication problems. Nobody would vouch for SQL Server either way. So "with memory" is not the safer choice for AD or SQL, and it may be the riskier one. That was my mistake, and you acted on it.

On this morning's failure, the error is worth reading closely. VM_BAD_POWER_STATE(..., halted, suspended) means XAPI wanted the VM halted and found it suspended. That is the same mismatch you hit at the top of this thread with "Start a copy", except it is now breaking the delta job that ran fine for months before you added the CR.

Two things would help pin it down: does it fail the same way tonight, and does the delta job succeed if you disable the CR job for a single run? If it does, that is a clean reproduction and worth a GitHub issue with this screenshot.

@Team-XO-Backend, this one now has an XAPI error and a clear before and after, so it has moved past being a docs question.

Dezerd

Thank you for that answer.
I will make the following changes:

Switch the snapshot to normal
Turn off replicated VMs
Lanch the job

Is there any documentation with details about the different snapshot modes (Normal, with memory, or offline)?

Thanks.

poddingue

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.

Dezerd

Unfortunately, I need to have a CR job that works because we have a constraint this weekend.
However, I'm available to provide you with logs if necessary.

I could let you know if changing the status of the replicated VM (from suspended to forced shutdown) has an impact on the backup chain. (with normal snapshot).

poddingue

Getting your backups working for the weekend is the right call.
If you do flip the replica from suspended to a forced shutdown, whatever you notice about the chain afterwards would be worth a line here.
That is the one question I could not answer, and I have not found it written down anywhere either.
On the logs, I would wait until someone from the storage side actually asks, so you are not gathering things nobody reads. The VM_BAD_POWER_STATE from post #3 is with @Team-Storage now, though I honestly do not know how fast that gets picked up.
Sorry again for the memory advice, and thanks for staying patient with a thread that changed direction twice on you.

Dezerd

So, as planned, I changed the snapshot to “normal” in the backup job and performed a forced shutdown on the replicated VMs.

This morning, the job status is “successful,” but two VMs (both of which have the ADDS role—this may be a coincidence) are showing this warning:

The type is still delta despite the warning message.

For alle the other VM, the backup seems ok.

tjkreidl

@Dezerd What does "xe task-list" show as far as active processes?

Dezerd

It's empty. Why ?