CR - Cannot start copy because suspended

Dezerd

Thank you for that answer.
I will make the following changes:

Switch the snapshot to normal
Turn off replicated VMs
Lanch the job

Is there any documentation with details about the different snapshot modes (Normal, with memory, or offline)?

Thanks.

poddingue

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.

Dezerd

Unfortunately, I need to have a CR job that works because we have a constraint this weekend.
However, I'm available to provide you with logs if necessary.

I could let you know if changing the status of the replicated VM (from suspended to forced shutdown) has an impact on the backup chain. (with normal snapshot).

poddingue

Getting your backups working for the weekend is the right call.
If you do flip the replica from suspended to a forced shutdown, whatever you notice about the chain afterwards would be worth a line here.
That is the one question I could not answer, and I have not found it written down anywhere either.
On the logs, I would wait until someone from the storage side actually asks, so you are not gathering things nobody reads. The VM_BAD_POWER_STATE from post #3 is with @Team-Storage now, though I honestly do not know how fast that gets picked up.
Sorry again for the memory advice, and thanks for staying patient with a thread that changed direction twice on you.

Dezerd

So, as planned, I changed the snapshot to “normal” in the backup job and performed a forced shutdown on the replicated VMs.

This morning, the job status is “successful,” but two VMs (both of which have the ADDS role—this may be a coincidence) are showing this warning:

The type is still delta despite the warning message.

For alle the other VM, the backup seems ok.

tjkreidl

@Dezerd What does "xe task-list" show as far as active processes?

Dezerd

It's empty. Why ?

tjkreidl

@Dezerd To check if there is perhaps a hung process that is interfering. I assume there is sufficient memory and storage space. Do any I/O disk errors show in /var/log/SMlog ? Just to clarify. are there VMs that do not show any such error condition?

poddingue

You did the experiment I asked for and then some, thank you.
One caveat before anyone reads too much into it: you changed two things at once, the snapshot mode AND the replica power state, so the "fell back to a full" on the two ADDS VMs cannot be pinned on the forced shutdown by itself, right?
The line I would look at in your screenshot is "failed to open disk" under Clean VM directory, since that step works on the existing chain rather than on the new transfer, and it fits a fallback to full better than the snapshot mode does.
@tjkreidl's SMlog question is the right next step and I would follow that rather than anything from me.
I still cannot tell you what the power-state change did to the chain.

Dezerd

I did indeed make both changes at the same time.

The status of the backup this morning is completely normal (even on ADDS VMs).

I think that the forced shutdown in order to correct the status error caused a problem during the next delta. Why only on ADDS VMs? I can’t explain it.

I had no errors on these VMs at that time.
(Especially since I have another full backup job that doesn’t cause any problems).