Large VM Migration fails, host has free disk - Storage_error

ditzy-olive

I am trying to migrate a rather large vm 1.6T of disks in total.
It runs for 15 Hours and then fails with There is insufficient space
The target xcp-ng however has 3.09TB of disk free so I am a bit confused where the insufficient space is coming from.
They hosts are running XCP-ng 8.2.1, other VM could be migrated without issue but struggling with this large one.
I could potentially start splitting the services from the VM into smaller VM which wouldn't be a bad idea anyway, but still would like to understand how to debug this.

"(((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1564))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
message	"INTERNAL_ERROR(Storage_error ([S(Internal_error);S(Storage_error ([S(Internal_error);S(Storage_error ([S(Internal_error);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_44);[S();S(There is insufficient space);S()]]]))]))]))]))"
name	"XapiError"
stack	"XapiError: INTERNAL_ERROR(Storage_error ([S(Internal_error);S(Storage_error ([S(Internal_error);S(Storage_error ([S(Internal_error);S(Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_44);[S();S(There is insufficient space);S()]]]))]))]))]))\n    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202512060836/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///opt/xo/xo-builds/xen-orchestra-202512060836/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202512060836/packages/xen-api/index.mjs:1073:24)\n    at file:///opt/xo/xo-builds/xen-orchestra-202512060836/packages/xen-api/index.mjs:1107:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202512060836/packages/xen-api/index.mjs:1097:12)\n    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202512060836/packages/xen-api/index.mjs:1270:14)

Pilow

@ditzy-olive Hi,

What are the type of SRs (Source and destination ?) NFS . iSCSI ? local or shared ?
What is the free space on both, before launching the migration ?

ditzy-olive

@Pilow both are local.
the host where it's being moved from has 2.3 TB free the host where it's being migrated to has 3TB Free.
I moved other VM without any issue.

Pilow

@ditzy-olive what bugs me is that error is happening 15h after the start

1.6Tb disks in total, you could detach disks, migrate VM+system, then migrate secondary disks and re attach them ?

or try offline migrating ? perhaps this VM is too i/o intensive and fills up its snapshots ?

source and destination SRs are thin or thick ?

nikade

Are you using EXT or LVM?
I've seen this as well with large VM's and not enough free space on the source host, the only solution is to either free up some space or do an offline copy. Migration wont work, so copy is your only option.

ditzy-olive

I managed to do the migration by splitting over the two storages the remove system has.
This made it work and I managed to move that one VM. However I still have a second one
It's thick provisioned (LVM - never can remember the terms thin/thick).
Offline isn't an option unfortunately since I need the services running, can't take if off for 15 hours. But since two of the services are docker containers, both having a separate disk of 500GB, I can make it work by setting up a two new systems and moving everything from the container there. Which would reduce the size by 1TB and make thing easier to manage.
Was hoping I could finally upgrade the main xcp-ng last weekend.