@iamsumeshks as said on reddit, it seems that you are good to go
monitor your nics bandwidth in hosts stats to ensure it goes on the good pif.

PREFNIC is the one where DRBD works i guess

@Bastien-Nollet many thanks

@Bastien-Nollet said in backup mail report says INTERRUPTED but it's not ?:

Ok so 1s is slightly not enough, thanks for the update.

Reply

still had issues with 2 sec delay.

it has now been 2 days with 3 sec delay, and no more. could be the sweet spot

@florent still have the issue
did @storage finally spawned after your summoning ?

@Bastien-Nollet another interrupted today.
I'll add one second by one second till I see it disappear.

just modified to 2s delay.

@Bastien-Nollet oopsy.

sadly, INTERRUPTED is back....

this is what I see in the backup report (a backup job of 12VMs to same S3 remote, only 1 interrupted...)

backup JOB in XOA is all green :

26 backups in a row without interruption, spanning 2 days
And i'm on the 1 second fix

guess it is enough...

@Bastien-Nollet said in backup mail report says INTERRUPTED but it's not ?:

I've done some more testing and looked at the code, and I wasn't able to reproduce this behaviour once. It's also unclear to me why it can happen.

I didn't tell but my Remotes are S3 Remotes... could it be because of that ?

@dcskinner make it a user préférence, and we get the best of both worlds

@Bastien-Nollet okay i'll do that tonight and will report back

@rtjdamen hey there,

did you ever modify in the POOL advanced tab at the bottom the BACKUP network & MIGRATION network ?

fiddle with these, note what they are actually set, and change them if possible
between each change, check your SR stats

@florent could explain better how SR stats use the backup network to get the RRDs...

I guess you have stalled RRD tasks too ? in tasks list ?

@Gheppy if you can have downtime snapshot/revert works

@wilsonqanda HAHA. problem resolved instantly for me on last SR I had the troubles

playing with the "rescan all ISO SRs" made the guest tools iso reappear

AND ALL MY OTHER VDIs ON THIS SR/HOST that were invisible

donnnnn't ask me HOW...

but this trick didn't work on other local RAID5 SR that have the problem

@wilsonqanda it's an SMB iso SR on my end, I think this is why it is not impacted

can't snapshot a file on Microsoft SMB share

@wilsonqanda strangely my ISOs were not impacted, except for the default XCP TOOLS iso that seems gone

the "base copy" regroup bug on homepage of the SR, and the orphan base copies in the DASHBOARD/HEALTH seems to be linked to same problem

we have much fast clones VMs depending on those base copies seen as orphan

we are narrowing on the problem, just need the dev to level it, have faith

i could be wrong but in

/usr/local/lib/node_modules/xo-server/dist/xapi-object-to-xo.mjs

I see a lot of

if (obj.is_a_snapshot || obj.$snapshot_of !== undefined)

seems to be a way of managing both "old version" and "new version" (or "broader version") to define if a VDI is a snapshot
there has been an evolution in the code at some point ?

and someone somewhere in recent updates (10 dec) forgot the || on some important action

my recent SR problem appeared when I was snapshoting/deleting VDI/reverting snapshot
all VMs suddenly didn't have visible VDIs anymore on the whole SR.

I could be wrong, but the fact that ALL VDIs are now seen as snapshot seems smelly with this ||

https://github.com/vatesfr/xen-orchestra/commit/85596da79217070bf4431135bbb5b0d2cf04e45b

0 spacotte-vates committed to vatesfr/xen-orchestra

fix(xo-server): improve handling of xapi snapshots (#9231)

In some cases, when converting an xapi object into an xo object, the xapi.is_a_snapshot param may be set to false even tho the xapi.snapshotOf refers to another xapi object. Handle this case better.

@wilsonqanda @anthoineb @olivierlambert

we are now impacted on a SR that didn't have the problem... now all disks are invisible and DASHBOARD/HEALTH show some strange dates on base copies...

halt/snap/revert correct the issue but needs to be done on each VM

all VDIs seem to be regrouped strangely on unique base copy

there is something seriously wrong happening

no impact on production of VMs though, just management issues in XOA web ui
even backups are OK.

halp.

but each remote (SOURCE and DESTINATION) can be attached to the same proxy... so, the proxy read/writes

and you have to manage your network accordingly so that this proxy can reach each remote (here goes the static routing or vpns...)

@fcgo this is where you reach a limit...
in an XOA backup copy job configuration, you select the proxy, and it drives source AND destination

you cannot have a proxy for SOURCE and a proxy for DESTINATION

@fcgo when adding an XOPROXY to a job, it flows from XOPROXY to the remote
I think XOA is not involved, checked the network bandwidth, xoa was sleeping

XOPROXY read/writes from source remote to destination remote

@fcgo hello there,

from my experience : XOA proxies are just some stripped downed XOA. no limitation on backup functionnalities.
I'm ok with you, it lacks from in depth informations on how the backups are done...

only place to find compression informations is in Disaster Recovey jobs.

we upgraded CPU & RAM of our XOA, but offloaded all backup tasks to proxies
if you have 1000+ VMs i think you already have at least 16Gb of RAM for dom0 (or you have 500 hosts with 2VMs each and stick with the default...)

assigning jobs to proxies is kind of manual (for exemple veeam can manage a pool of proxies and take the least occupied or chosen one(s) in the pool)

if you implement proxies, you will be confronted to assign them to a job AND subsenquently assign them to remotes too ! you have to have a good planification of what you want to be done (a remote locked to a proxy is not seen by other proxies... sometimes need to create the SAME remote twice to attach it to two proxies... and be sure not to run in parallele on these two...)

planification of backups of 1000VMs is something.