Our future backup code: test it!

Andrew

@florent @olivierlambert Not good!... I reverted back due to problems.

Replication does run. It seems to work. But it leaves lots of zombie export tasks, like:
Exporting content of VDI ftp_root_jgfkd through NBD+CBT (on xcp1) 17%

I get about 80% stuck tasks. A toolstack restart clears the tasks....
CR settings: CBT enabled. NBD=2. Purge enabled.

flakpyro

@Andrew I am having the exact same issue! Also rolling back! Glad its not just me haha

Andrew

@flakpyro @florent I had the same problem with the test branch that did not get solved.

florent

thabnks for the test
are you doing replication only ? or is a job doiing replication and backup ?

flakpyro

@florent I am just running a "Delta Backup" to an NFS SR. I haven't rolled back yet so if there is any sort of logs i can provide let me know. This is my "home lab" at home where i test new versions of XOA and XCP-NG updates before we end up seeing new features in production with our full XOA appliance at work.

Andrew

@florent The job is just Continuous Replication to another local host. These were normal hourly delta transfers.

Tristis Oris

Same. Delta, 2 threads, 2 runing, but jobs not cleared.

Tristis Oris

Continuous Replication failed with Error: HANDLE_INVALID(VDI, OpaqueRef:db811593-f18e-7d15-ea8a-465bf7a94917)

manilx

@florent https://xcp-ng.org/forum/topic/10857/suddenly-backup-ends-but-tasks-remain-in-task-list/4

Rolled back and will stay there until this is fixed. Gave me heart attack....

florent

PR for unclosd task should reach master today
https://github.com/vatesfr/xen-orchestra/pull/8590

@Tristis-Oris vould you share ( by direct message if you prefer ) the rapport of the failed replication ?

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(@xen-orchestra/xapi): really stop xapi task #8590

Tristis Oris

@florent https://pastebin.com/vhLXZ94F

florent

@Tristis-Oris thanks, but it is marked as a private pastebin

Tristis Oris

@florent weird.. it say Pending Moderation. log is too huge to post here.

check pm. it should work.

Andrew

@florent On a quick CR test, fix_nbd_progress works correctly for me.

Andrew

@florent I have the current master (9a833) running now and it's working so far. No change in speed at this time (just testing CR delta).

I did notice that the active tasks show only one Export/NBD but I have 4 configured. I don't know if this is a task view issue (I guess I only need to see one) or a true 1x NBD export.

flakpyro

@florent Installed in the same test environment as last week, this time the backup ran successfully and the tasks cleared as usual. I will keep running this build over the next week but so far so good! I should note i am only running Delta backups at home. We use Delta + CR jobs in production at work but that is running full blown XOA. Is the plan to push this to XOA in the next monthly release?

Tristis Oris

Need to restart toolstack on each host after fix. Now all jobs dissapeared.

manilx

@florent Just updated XO and it ran my nightly backup without issues clearing tasks.
Nicely done.

florent

I am happy to hear this . I am working on this code almost full time since december

Did you try a restore / file restore to ensure the backups are exactly as intended ?

Tristis Oris

still not fixed.