@Biggen You can do this via cli...see if this doc is helpful (just make sure to back/move/copy any data before performing these steps): https://support.citrix.com/article/CTX131328

@olivierlambert Thanks, I tried running fstrim but it returned "the discard operation is not supported" on this NFS SR mount. I also tried running on SSD mounts which do support TRIM and they all returned the same message about "discard...not supported."

The VM OS (CentOS 7.8) does support TRIM in kernel so not sure if xcp-ng or xenserver is the reason why the VM can't issue fstrim on the NFS SR mount or the SSD SR mounts.

@nraynaud Hi, Thank you for your assistance. Here is /etc/fstab from the VM which is CentOS 7.x:

UUID=ef4eef1c-dea6-452e-9fb4-20948465f484 /boot                xfs    defaults        0 0
/dev/mapper/centos-root /                                      xfs    defaults        0 0
/dev/xvdh1  /emails                                            xfs    defaults        0 0
/dev/mapper/databases-var_lib /var/lib                         xfs    defaults        0 0
/dev/mapper/sas10k-var_log /var/log                            xfs    defaults        0 0
/dev/mapper/websites-var_www /var/www                          xfs    defaults        0 0
/dev/mapper/centos-swap swap                                   swap   defaults        0 0

UUID=20706a6e-bda3-4277-abf1-c74c4619a1d8 /Backups/      xfs    defaults        0 0

And this is from XO:

Hi @olivierlambert , sorry the VHD path I provided above was wrong.

It appears the SR is showing it's using 421 GB while the VM OS shows 1.4 GB.

How long does it take for "VHD to free blocks on the filesystem"? This is NFS SR.

Thx,

SW

EDIT: @olivierlambert Thank you, I checked the SR and it's showing much larger size (421 GB) while the VM OS shows only 1.3 GB:

root@freenas25[/mnt/Tank/VMs]# du -hs 6e07202c-38f0-dece-52d6-d81743635645/55a93b7d-1e2e-4903-961b-c3df8b73fee3
.vhd
421G    6e07202c-38f0-dece-52d6-d81743635645/55a93b7d-1e2e-4903-961b-c3df8b73fee3.vhd

VM OS

[root@server420 ~]# du -hs /Backups/
1.3G    /Backups/

@olivierlambert Thanks for your help:

• xo-server 5.70.0

• xo-web 5.74.0

• XCP-ng 8.0

Hi,

I'm running into an issue where the backup jobs are taking longer than 24 hrs and failing. I understand this is a limitation with xenserver (per this post).

However after further investigation I'm seeing that the initial Delta & Replication are copying the entire drive even though they are on thin provisioning.

For example, on a VM, I have a 500GB drive that is barely used (1.3 GB):

/dev/xvdg1               500G  1.3G  499G   1% /Backups

The above drive is provisioned from a FreeNas which is provisioned as "thin":

But when XO is running the backup job (Delta & Continous), it's moving 500 GB of data across the network and storing the entire 500 GB of the Delta Backup and Continuous Replication as part of the initial full backup.

I thought with thin provisioning, the backups would only be of actual used data (in my example above) that would be 1.3 GB instead of 500GB of empty data.

Am I doing something wrong?

The backup job has been running for almost 22 hrs and appears to be copying the entire 500GB drive even though it's mostly blank:

On the target remote for the Delta Backup, I see it still copying data:

root@freenas25[~]# ls -alh /mnt/Tank/VMs/Backups/xo-vm-backups/3f9a9ffe-3952-c9fb-4481-8aa017c
708a3/vdis/ef24e62c-8829-4956-9aa6-10ebb3dd0e8f/55a93b7d-1e2e-4903-961b-c3df8b73fee3
total 325168553
drwxr-xr-x  2 4294967294  wheel     3B Nov  5 13:32 .
drwxr-xr-x  8 4294967294  wheel     8B Nov  5 13:32 ..
-rw-r--r--  1 4294967294  wheel   310G Nov  6 10:41 .20201105T183221Z.vhd

backup-job-log.txt

Ok, was able to force shutdown via CLI and now I'm able to start VM on source server:

# xe vm-reset-powerstate vm=<vm name> --force

I'm not going to perform live migrations anytime soon again. I was trying to move VMs so we can do an upgrade which requires wiping out all storage because we used experimental ext4.

When I login to xcp-ng cli, I see this when I run list_domains where the UUID of this VM looks different cli vs what xo shows:

@olivierlambert Tried Force shutdown, Force reboot and both return similar error about "VM does not exist in xenopsd":

vm.stop
{
  "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c",
  "force": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "d085e780-7fc9-1e2d-60f1-ec8c144b4cfa",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20201030T18:30:41Z",
    "finished": "20201030T18:30:41Z",
    "status": "failure",
    "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
}

@olivierlambert Tried different combinations of disabling source, target and both. The only way I can see the VM in XO is by enabling the source server.

I even tried removing the VM from the source server so I can restore from backup and I'm getting error:

vm.delete
{
  "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "66366c1a-e0fa-cc12-0ecf-aab05f558613",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20201030T18:23:39Z",
    "finished": "20201030T18:23:39Z",
    "status": "failure",
    "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
}