We'll re-enable it when we'll find a better solution.

Thank you all for your reports

0 julien-f committed to vatesfr/xen-orchestra

fix(backups): disable VHD stream validation by default

Introduced by 68b2c287e

Fixes https://xcp-ng.org/forum/post/61118

Current code is too heavy on the main thread, disable by default until a better solution is fixed.

I too noticed by backup times went sky high recently. I just made the suggested switch in the config and that is the difference between before/after. My box has 2x E5-2697a v4 processors.

Transfer without setting

Transfer with setting

• when nothing is set, XOCE makes the transfer using a single vCPU to the maximum and the rest up to 5% max . The maximum transfer being up to 23Mb/s
• when the setting exists, XOCE uses all 12 vCPUs between 30% and 45%. The transfer is between 145Mb/s and 350Mb/s.

In both cases, the transfer is done on a 10Gib LAN and I use http mod for connecting to Servers

Regarding the perf issue, I will work on it as soon as I'm able to reproduce on my side

If the impact is too big, we'll disable it by default…

0 julien-f committed to vatesfr/xen-orchestra

fix(backups/importDeltaVm): handle `suspend_VDI === 'OpaqueRef:NULL'`

Fixes https://xcp-ng.org/forum/post/61169

Introduced by 4d55c5ae4

@Andrew said in Slow backups (updated XO source issue):

@julien-f The time it takes for CR to run seems to be back to normal with the added config

[backups.vm.defaultSettings]
validateVhdStreams = false

I have Concurrency set to 4.

There's more than enough local ethernet bandwidth between machines and XO is running on the pool master. There's lots of cores and memory for XO. But with validateVhdStreams = true there seems to be some single threaded issues with node because it uses 90%-120% cpu.

With validateVhdStreams = false the CPU usage is about 110%-150%, so still some single threaded issues but a lot less as node is able to use more than one core (but not all of them).

Also... I see the warning that appeared from an earlier commit but does not cause problems:

 transfer
 Suspend VDI not available for this suspended VM

It has no consequences except from this message, I'll fix it ASAP

julien-f opened this pull request in vatesfr/xen-orchestra

closed fix(backups): restore VM with memory as suspended #6774

[backups.vm.defaultSettings]
validateVhdStreams = false

I have Concurrency set to 4.

There's more than enough local ethernet bandwidth between machines and XO is running on the pool master. There's lots of cores and memory for XO. But with validateVhdStreams = true there seems to be some single threaded issues with node because it uses 90%-120% cpu.

With validateVhdStreams = false the CPU usage is about 110%-150%, so still some single threaded issues but a lot less as node is able to use more than one core (but not all of them).

Also... I see the warning that appeared from an earlier commit but does not cause problems:

 transfer
 Suspend VDI not available for this suspended VM

I have been getting a lot of errors (xcp7 is the destination of the CR) too:

root@xcp7 Xapi#putResource /import_raw_vdi/ Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
    at new NodeError (node:internal/errors:399:5)
    at ClientRequest.onclose (node:internal/streams/end-of-stream:154:30)
    at ClientRequest.emit (node:events:525:35)
    at ClientRequest.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202304161246/@xen-orchestra/log/configure.js:52:17)
    at TLSSocket.socketCloseListener (node:_http_client:465:9)
    at TLSSocket.emit (node:events:525:35)
    at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202304161246/@xen-orchestra/log/configure.js:52:17)
    at node:net:322:12
    at TCP.done (node:_tls_wrap:588:7)
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 'ERR_STREAM_PREMATURE_CLOSE'
}

You can disable this check by setting the following entry in your xo-server/xo-proxy config:

[backups.vm.defaultSettings]
validateVhdStreams = false

If you can it would be interesting if you could provide the duration of the backup job and the CPU usage on your XO appliance with and without the validation.
And please also provide the transferred size of the backup to ensure the difference does not come from here (an error I could very likely make ^^)

Also, we've just removed some other checks which where no longer necessary with this change: https://github.com/vatesfr/xen-orchestra/pull/6773

julien-f opened this pull request in vatesfr/xen-orchestra

closed Revert "feat(backups/writeVhd): check file has expected size (#6703)" #6773