backup tasks (exporting VDI) stopped showing progression, bug introduced on build master 7994f or shortly before

manilx

Was: backup tasks stopped showing after moving VM disks to shared NFS SR

Hi

I have my VM's running on 2 Protectli hosts. I have used a shared QNAP NFS SR as storage for backups.

Migrating VM's between hosts signified also moving the disks, taking longer than it would be if the disks were on shared storage.
Also RPU is not possible.

So I moved all VM disks to shared NFS storage. No problem there, all running fine.

BUT when backups are running and I look at the tasks I see:
The snapshot being created and then for a second the disk backup beginning BUT where I would normally see the progressing taks with percentage nothing is shown. Blank.
This while the tasks are clearly running as the backup progresses (network stats on XO VM and hosts showing progression and network/disk stats on the NFS host).
The backup tasks just finish fine.

UPDATE: Actually it had nothing to do with the move to shared storage but it's an issue with XO

Reverted back to commit d8370: same issue
Trying to go back further if I can pinpoint when this started....

manilx

commit d8370: same
commit e4959: same
commit 1a7b5: shows exporting VDI percentage just fine (was the last time I actively monitored backup because of https://xcp-ng.org/forum/post/94267)
commit 7994f: shows exporting VDI percentage just fine

Sp between 7994f and e4959 something was introduced that causes this.

Staying on 7994f for the time being

flakpyro

@manilx I am seeing this as well (on 2effd)

manilx

@flakpyro Spent the better part of my Saturday on this because I was looking in the wrong direction, when it is just a newly introduced bug....

florent

@manilx
can you retry with NBD enabled ? I have a hunch that it is related to non NBD backup, and the changes introduced to handle qcow2
(you'll need to restart xo to stop the task )

my bet is thaht the breaking commit are these https://github.com/vatesfr/xen-orchestra/commit/cbc07b319fea940cab77bc163dfd4c4d7f886776 and https://github.com/vatesfr/xen-orchestra/commit/bb4e14b6ddfe7d682c4bdea089c8407caebfb9a5

0

fbeauchamp committed to vatesfr/xen-orchestra

feat(xapi): support import and export of qcow2 disks (#8506)

This PR implement a limited support of generating and reading qcow2 . The support is limited since it only handles qcow2 with 64KB cluster size, with this structure :

    header
    refcount table
    refcount blocks
    L1 table
    L1 tables
    data clusters

generated by https://github.com/qemu/qemu/blob/a9cd5bc6399a80fcf233ed0fffe6067b731227d8/scripts/qcow2-to-stdout.py

0

fbeauchamp committed to vatesfr/xen-orchestra

fix(backups): use the right format (qcow2/vhd) (#8750)

introduced by https://github.com/vatesfr/xen-orchestra/commit/cbc07b319fea940cab77bc163dfd4c4d7f886776

manilx

@florent I always have/had NBD enabled!

flakpyro

@florent i have NBD enabled. Whats strange is i updated builds twice last week and backups continued to run. (Though progress in the tasks view disappeared as pointed out in this thread).

Last night we had a storm that knocked power out and my UPS's ran out of juice taking my home lab down, when everything came back up after backups would no longer run and would show ""invalid HTTP header in response body". I rolled back to a earlier commit (c7167 June 5th) and now not only are backups working again but progress in the tasks view is also displaying properly again.

Edit: Moved to commit 7994f from June 23rd and i still see progress there as well.

flakpyro

This seems to be fixed in the latest commit and also does not happen in the just released XOA 5.108.0

manilx

fixed in commit fcefa