Posts made by acomav

@olivierlambert Hi Olivier. I'll see what I can do. I've spent the weekend cleaning up my backups and catching up on mirror transfers. Once completed, I'll do a few custom backups at various nconnect values.

Just replying to thank you for pointing this out. I have been having very poor backup speeds for over a month and this sorted it out.
I have only used nconnect=4 and 6 for my NFS shares.

@olivierlambert
I can confirm it was my side. I had to do a few things to get the VMware Virtual disks to free up empty space and once I did, the VM Import to XCP-NG to an NFS SR successfully copied the virtual disk in a thin mode.
For anyone reading this who will be preparing to jump ship off VMware.

I am using vSphere 6.7. I have not tested against vSphere 7 yet. Not bothering with vSphere 8 for obvious reasons. My VM was a CentOS 7 VM with LVM to manage the 3 virtual disks.

1. Make sure you Virtual Hardware is at least version 11. My test VM was a very old one still on version 8.
2. For the ESXi host the VM lives on (but you should probably go all hosts in the cluster), go into Advanced settings, and enable (change 0 to 1) VMFS3.EnableBlockDelete. I thought I had this enabled but only 2 of the 5 hosts in the cluster did. You may need to check this is not reset after updates.
3. Due to using CentOS 7 (perhaps) I could not used 'fstrim' with the discard mount option. It was not supported. I filled up the diskspace with zeros, synced, and then removed the zeroes.

# cd /mount point; dd if=/dev/zero of=./zeroes bs=1M count=1024; sync; rm zeroes

Change count=1024 (Which will create 1 GB of zeroes in a file) to however big a file you require to nearly fill up the partition / volume. eg count=10240 will make a 10 GB file.
Windows users can use 'sdelete'.

I could have waited for vSphere to automatically clean up the datastore in the background at this stage, but I was impatient and 'storage motioned' the virtual disks to NFS storage in Thin mode. I confirmed only the used space was copied across. I then migrated the disks back to my HP Nimble SAN and retained thin provisioning.

@olivierlambert Hi.
The disk sizes (and vmdk file size) are 150GB and 170GB. Both are in a Volume group and one Logical Volume using 100% of the Volume group mounted using XFS.

Disk space in use is 81%:

# pvs
  PV         VG         Fmt  Attr PSize    PFree
  /dev/sda2  centos     lvm2 a--   <15.51g    0 
  /dev/sdb   VolGroup01 lvm2 a--  <150.00g    0 
  /dev/sdc   VolGroup01 lvm2 a--  <170.00g    0 

# vgs
  VG         #PV #LV #SN Attr   VSize   VFree
  VolGroup01   2   1   0 wz--n- 319.99g    0 
  centos       1   2   0 wz--n- <15.51g    0

# lvs
  LV        VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  IMAPSpool VolGroup01 -wi-ao---- 319.99g 

# df -h
/dev/mapper/VolGroup01-IMAPSpool  320G  257G   64G  81% /var/spool/imap

The vmdk files live on an HPE/Nimble CS3000 (Block iscsi). I am now thinking I will need to get into the VM and free up discarded/deleted blocks....which would make the vmdk sizes smaller. (as they are set to thin provisioned with vmfs)
I'll do that and retry and report back if I see the the full disk being written out to XCP-NG.

@florent The VM is on an NFS SR which is thin provisioned. LVM is inside the VM on the virtual disks.

Hi, a question about these patches and thin provisioning.

My test import now works, however, it fully provisioned the full size of the disk on an NFS SR.

[root@XXXX ~]# ls -salh /mnt/NFS/d8ad046d-c279-5bd6-8ed7-43888187f188/
total 540G
4.0K drwxr-xr-x  2 root root 4.0K Feb  6 09:33 .
4.0K drwxr-xr-x 27 root root 4.0K Feb  1 21:22 ..
151G -rw-r--r--  1 root root 151G Feb  6 10:45 1c3b93da-de07-4a4f-8229-60635bc2f279.vhd
 13G -rw-r--r--  1 root root  13G Feb  6 09:43 1eae9130-e6eb-45be-ae25-a7dcb7ee8f4e.vhd
171G -rw-r--r--  1 root root 171G Feb  6 10:51 751b7a5f-df32-4cb1-9479-e196671e7149.vhd

The two large disks are in an LVM VG on the source and combined, use up 253 GB of the 320 GB LV. They are thin provisioned on the VMware side.

Am I wrong to expect the vhd files on the NFS SR to be smaller than what I see? Does LVM on the source negate thin provisioning on the xcp-ng side?

Not a big deal, I am just curious.

Thanks

@acomav

Import completed. Great work @florent.

@florent
Thanks. I have kicked off an Import but it takes 2 hours however....the first small virtual disk has now been successful whereas it was failing, so I am confident the rest will work. Will update then.

Thanks

I patched my XO source VM with the latest from 5th Feb and still had the same error.
"stack": "Error: no opaque ref found in undefined

It may be I am not patching correctly so I have added a XOA trial and moved to the 'latest' channel and have ping @florent with a support tunnel to test in the morning.

@acomav
Replying to myself.

I redid the job with a snapshot from a running VM to a local SR. Same issue occurred at the same time.

I came across the same error today before seeing this thread. Importing a 3 disk VM (powered off).
The first smaller disk failed first.
I saw the post about the patch and applied to my XO source VM. (Ronivay Debian image with the disk extended to 30 GB).
I then tried a live (with snapshot) 10 GB 1 disk VM to local thick LVM SR, and it was successful.
I retried the big VM to a NFS SR and it failed in the same spot.

Feb 03 10:41:42 xo-ce xo-server[2902]: 2024-02-03T10:41:42.888Z xo:xo-server WARN possibly unhandled rejection {
Feb 03 10:41:42 xo-ce xo-server[2902]:   error: Error: already finalized or destroyed
Feb 03 10:41:42 xo-ce xo-server[2902]:       at Pack.entry (/opt/xo/xo-builds/xen-orchestra-202402030246/node_modules/tar-stream/pack.js:138:51)
Feb 03 10:41:42 xo-ce xo-server[2902]:       at Pack.resolver (/opt/xo/xo-builds/xen-orchestra-202402030246/node_modules/promise-toolbox/fromCallback.js:5:6)
Feb 03 10:41:42 xo-ce xo-server[2902]:       at Promise._execute (/opt/xo/xo-builds/xen-orchestra-202402030246/node_modules/bluebird/js/release/debuggability.js:384:9)
Feb 03 10:41:42 xo-ce xo-server[2902]:       at Promise._resolveFromExecutor (/opt/xo/xo-builds/xen-orchestra-202402030246/node_modules/bluebird/js/release/promise.js:518:18)
Feb 03 10:41:42 xo-ce xo-server[2902]:       at new Promise (/opt/xo/xo-builds/xen-orchestra-202402030246/node_modules/bluebird/js/release/promise.js:103:10)
Feb 03 10:41:42 xo-ce xo-server[2902]:       at Pack.fromCallback (/opt/xo/xo-builds/xen-orchestra-202402030246/node_modules/promise-toolbox/fromCallback.js:9:10)
Feb 03 10:41:42 xo-ce xo-server[2902]:       at writeBlock (file:///opt/xo/xo-builds/xen-orchestra-202402030246/@xen-orchestra/xva/_writeDisk.mjs:9:22)
Feb 03 10:41:42 xo-ce xo-server[2902]: }
Feb 03 10:41:45 xo-ce xo-server[2902]: root@10.1.4.10 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
Feb 03 10:41:45 xo-ce xo-server[2902]:     at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xen-api/_XapiError.mjs:16:12)
Feb 03 10:41:45 xo-ce xo-server[2902]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xen-api/_getTaskResult.mjs:11:29)
Feb 03 10:41:45 xo-ce xo-server[2902]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xen-api/index.mjs:1006:24)
Feb 03 10:41:45 xo-ce xo-server[2902]:     at file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xen-api/index.mjs:1040:14
Feb 03 10:41:45 xo-ce xo-server[2902]:     at Array.forEach (<anonymous>)
Feb 03 10:41:45 xo-ce xo-server[2902]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xen-api/index.mjs:1030:12)
Feb 03 10:41:45 xo-ce xo-server[2902]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xen-api/index.mjs:1203:14) {
Feb 03 10:41:45 xo-ce xo-server[2902]:   code: 'IMPORT_ERROR',
Feb 03 10:41:45 xo-ce xo-server[2902]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
Feb 03 10:41:45 xo-ce xo-server[2902]:   call: undefined,
Feb 03 10:41:45 xo-ce xo-server[2902]:   url: undefined,
Feb 03 10:41:45 xo-ce xo-server[2902]:   task: task {
Feb 03 10:41:45 xo-ce xo-server[2902]:     uuid: 'e1ed657e-165c-0a78-2b72-3096b0550fed',
Feb 03 10:41:45 xo-ce xo-server[2902]:     name_label: '[XO] VM import',
Feb 03 10:41:45 xo-ce xo-server[2902]:     name_description: '',
Feb 03 10:41:45 xo-ce xo-server[2902]:     allowed_operations: [],
Feb 03 10:41:45 xo-ce xo-server[2902]:     current_operations: {},
Feb 03 10:41:45 xo-ce xo-server[2902]:     created: '20240203T10:32:22Z',
Feb 03 10:41:45 xo-ce xo-server[2902]:     finished: '20240203T10:41:45Z',
Feb 03 10:41:45 xo-ce xo-server[2902]:     status: 'failure',
Feb 03 10:41:45 xo-ce xo-server[2902]:     resident_on: 'OpaqueRef:e44d0112-ac22-4037-91d3-6394943789fd',
Feb 03 10:41:45 xo-ce xo-server[2902]:     progress: 1,
Feb 03 10:41:45 xo-ce xo-server[2902]:     type: '<none/>',
Feb 03 10:41:45 xo-ce xo-server[2902]:     result: '',
Feb 03 10:41:45 xo-ce xo-server[2902]:     error_info: [
Feb 03 10:41:45 xo-ce xo-server[2902]:       'IMPORT_ERROR',
Feb 03 10:41:45 xo-ce xo-server[2902]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
Feb 03 10:41:45 xo-ce xo-server[2902]:     ],
Feb 03 10:41:45 xo-ce xo-server[2902]:     other_config: { object_creation: 'complete' },
Feb 03 10:41:45 xo-ce xo-server[2902]:     subtask_of: 'OpaqueRef:NULL',
Feb 03 10:41:45 xo-ce xo-server[2902]:     subtasks: [],
Feb 03 10:41:45 xo-ce xo-server[2902]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 92)))'
Feb 03 10:41:45 xo-ce xo-server[2902]:   }
Feb 03 10:41:45 xo-ce xo-server[2902]: }
Feb 03 10:41:45 xo-ce xo-server[2902]: 2024-02-03T10:41:45.956Z xo:api WARN admin@admin.net | vm.importMultipleFromEsxi(...) [9m] =!> Error: no opaque ref found

I'm going to try the next test from the same (3 disk VM) but have it powered up with a snapshot and save to local LVM SR.

The XO error for that disk that failed.

     {
          "id": "38jiy3bsy5r",
          "properties": {
            "name": "Cold import of disks scsi0:0"
          },
          "start": 1706956341748,
          "status": "failure",
          "end": 1706956905772,
          "result": {
            "message": "no opaque ref found",
            "name": "Error",
            "stack": "Error: no opaque ref found\n    at importVm (file:///opt/xo/xo-builds/xen-orchestra-202402030246/@xen-orchestra/xva/importVm.mjs:28:19)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at importVdi (file:///opt/xo/xo-builds/xen-orchestra-202402030246/@xen-orchestra/xva/importVdi.mjs:6:17)\n    at file:///opt/xo/xo-builds/xen-orchestra-202402030246/packages/xo-server/src/xo-mixins/migrate-vm.mjs:260:21\n    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202402030246/@vates/task/index.js:158:22)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202402030246/@vates/task/index.js:141:20)"
          }
        },

Thanks @olivierlambert.
The problem is that one disk was already there running a PBX so I could not shut it off. Also, for reasons during its migration from Oracle Cloud to xcp-ng, the virtual disk finished up as /dev/xvdb for boot and OS. Adding a 2nd disk did not give me an option of where to place the disk and it was added as xvda. When I did a fast snapshot clone of the running VM, the cloned VM wanted to boot off /dev/xvda.
Anyway, thanks for your response and I look forward to seeing the XOA's evolution and this feature in the coming releases.

cheers

Hi,
Apologies if this has been answered, but I could not find what I was after when searching.

In XOA, is there a way in the interface to adjust the device position of a (2nd or 3rd) virtual disk of a VM. I had to resort to XCP-NG Centre to make the change I was after.

Here is a Xenserver description of where I went:
https://docs.xenserver.com/en-us/xencenter/8-2/vms-storage-properties.html

Thanks