CR - Cannot start copy because suspended

poddingue

On the documentation, I went looking and I do not think it exists.
The modes get named but never explained. The closest thing is the backup modifier tags section, which tells you how to override the mode for one VM with a tag and assumes you already know what the modes are: https://docs.xen-orchestra.com/xo5/backups#backup-modifier-tags. That is a gap on our side rather than something you missed, and it is written down now.

One thing before you make those changes. Switching to normal snapshots and powering off the replicas will probably make this morning's failure impossible to reproduce, and nobody has looked at it yet. If you can spare one more run in the current configuration, the VM_BAD_POWER_STATE on a delta job that worked for months is the interesting part.

If you would rather just get your backups working, do that instead. You have already spent enough of your week on this.

Also, correcting myself again: I tagged Team-XO-Backend earlier and that was the wrong team. A XAPI error should go to the storage side, because XO only calls into XAPI rather than implementing it. @Team-Storage, if anyone has a moment for the error in the screenshot at post #3.

Dezerd

Unfortunately, I need to have a CR job that works because we have a constraint this weekend.
However, I'm available to provide you with logs if necessary.

I could let you know if changing the status of the replicated VM (from suspended to forced shutdown) has an impact on the backup chain. (with normal snapshot).

poddingue

Getting your backups working for the weekend is the right call.
If you do flip the replica from suspended to a forced shutdown, whatever you notice about the chain afterwards would be worth a line here.
That is the one question I could not answer, and I have not found it written down anywhere either.
On the logs, I would wait until someone from the storage side actually asks, so you are not gathering things nobody reads. The VM_BAD_POWER_STATE from post #3 is with @Team-Storage now, though I honestly do not know how fast that gets picked up.
Sorry again for the memory advice, and thanks for staying patient with a thread that changed direction twice on you.

Dezerd

So, as planned, I changed the snapshot to “normal” in the backup job and performed a forced shutdown on the replicated VMs.

This morning, the job status is “successful,” but two VMs (both of which have the ADDS role—this may be a coincidence) are showing this warning:

The type is still delta despite the warning message.

For alle the other VM, the backup seems ok.

tjkreidl

@Dezerd What does "xe task-list" show as far as active processes?

Dezerd

It's empty. Why ?

tjkreidl

@Dezerd To check if there is perhaps a hung process that is interfering. I assume there is sufficient memory and storage space. Do any I/O disk errors show in /var/log/SMlog ? Just to clarify. are there VMs that do not show any such error condition?

poddingue

You did the experiment I asked for and then some, thank you.
One caveat before anyone reads too much into it: you changed two things at once, the snapshot mode AND the replica power state, so the "fell back to a full" on the two ADDS VMs cannot be pinned on the forced shutdown by itself, right?
The line I would look at in your screenshot is "failed to open disk" under Clean VM directory, since that step works on the existing chain rather than on the new transfer, and it fits a fallback to full better than the snapshot mode does.
@tjkreidl's SMlog question is the right next step and I would follow that rather than anything from me.
I still cannot tell you what the power-state change did to the chain.

Dezerd

I did indeed make both changes at the same time.

The status of the backup this morning is completely normal (even on ADDS VMs).

I think that the forced shutdown in order to correct the status error caused a problem during the next delta. Why only on ADDS VMs? I can’t explain it.

I had no errors on these VMs at that time.
(Especially since I have another full backup job that doesn’t cause any problems).

poddingue

Good news that it sorted itself out. Both changes went in at once and the next run came back clean on the ADDS VMs too, so I don't think there's anything left to pin down here, and I'd rather say that than invent an explanation after the fact.
Your question about the snapshot modes was the useful thing to come out of this thread.
I went looking, they genuinely aren't documented anywhere, and that's written up on our side now
If the fall back to a full ever comes back on the ADDS VMs specifically, that would deserve its own thread with the SMlog output @tjkreidl asked for, since nobody got to look at that part.