https://xcp-ng.org/docs/api.html
Restarting the API
Sometimes, you might want to restart the toolstack (API and storage stack). You can do it with the following command from the host console (or directly with Xen Orchestra):
xe-toolstack-restart
TIP
Restarting XAPI won't affect any running VMs. However, all backup/export tasks will be removed.
https://xcp-ng.org/docs/api.html
Restarting the API
Sometimes, you might want to restart the toolstack (API and storage stack). You can do it with the following command from the host console (or directly with Xen Orchestra):
xe-toolstack-restart
TIP
Restarting XAPI won't affect any running VMs. However, all backup/export tasks will be removed.
Switched to latest channel:
Current version: 5.99.1 - XOA build: 20241004
exact same error for me as i posted before:
Oct 08 04:12:21 xoa xo-server[152702]: 2024-10-08T09:12:21.139Z xo:backups:worker INFO starting backup
Oct 08 04:12:27 xoa xo-server[152702]: 2024-10-08T09:12:27.213Z xo:xapi:vdi INFO found changed blocks {
Oct 08 04:12:27 xoa xo-server[152702]: changedBlocks: <Buffer a0 11 60 00 00 00 00 60 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 511950 more bytes>
Oct 08 04:12:27 xoa xo-server[152702]: }
Oct 08 04:12:41 xoa xo-server[152702]: 2024-10-08T09:12:41.036Z xo:xapi:vdi INFO found changed blocks {
Oct 08 04:12:41 xoa xo-server[152702]: changedBlocks: <Buffer 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 102353 more bytes>
Oct 08 04:12:41 xoa xo-server[152702]: }
Oct 08 04:12:43 xoa xo-server[152702]: 2024-10-08T09:12:43.021Z xo:xapi:vdi INFO OpaqueRef:0c5189b6-8ef7-4d55-80c9-0aed7b165e18 has been disconnected from dom0 {
Oct 08 04:12:43 xoa xo-server[152702]: vdiRef: 'OpaqueRef:8b3097a7-0038-475f-a45e-5d386f3e3e26',
Oct 08 04:12:43 xoa xo-server[152702]: vbdRef: 'OpaqueRef:0c5189b6-8ef7-4d55-80c9-0aed7b165e18'
Oct 08 04:12:43 xoa xo-server[152702]: }
Oct 08 04:23:43 xoa xo-server[152702]: 2024-10-08T09:23:43.779Z xo:xapi:vdi INFO OpaqueRef:3ce8200f-f3ca-4548-b1b3-beac8b19ae2b has been disconnected from dom0 {
Oct 08 04:23:43 xoa xo-server[152702]: vdiRef: 'OpaqueRef:75005d3e-4d4c-4799-ab70-ee7e7a4f7e23',
Oct 08 04:23:43 xoa xo-server[152702]: vbdRef: 'OpaqueRef:3ce8200f-f3ca-4548-b1b3-beac8b19ae2b'
Oct 08 04:23:43 xoa xo-server[152702]: }
Oct 08 04:24:59 xoa xo-server[152702]: 2024-10-08T09:24:59.120Z xo:xapi:vdi INFO OpaqueRef:98b20faa-2c5e-4b5a-9946-63b71857f200 has been disconnected from dom0 {
Oct 08 04:24:59 xoa xo-server[152702]: vdiRef: 'OpaqueRef:89cedb32-81c9-482c-b6b0-e9d0102fd198',
Oct 08 04:24:59 xoa xo-server[152702]: vbdRef: 'OpaqueRef:98b20faa-2c5e-4b5a-9946-63b71857f200'
Oct 08 04:24:59 xoa xo-server[152702]: }
Oct 08 04:24:59 xoa xo-server[152702]: 2024-10-08T09:24:59.980Z xo:xapi WARN retry {
Oct 08 04:24:59 xoa xo-server[152702]: attemptNumber: 0,
Oct 08 04:24:59 xoa xo-server[152702]: delay: 5000,
Oct 08 04:24:59 xoa xo-server[152702]: error: XapiError: VDI_IN_USE(OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e, destroy)
Oct 08 04:24:59 xoa xo-server[152702]: at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Oct 08 04:24:59 xoa xo-server[152702]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
Oct 08 04:24:59 xoa xo-server[152702]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1041:24)
Oct 08 04:24:59 xoa xo-server[152702]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1075:14
Oct 08 04:24:59 xoa xo-server[152702]: at Array.forEach (<anonymous>)
Oct 08 04:24:59 xoa xo-server[152702]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1065:12)
Oct 08 04:24:59 xoa xo-server[152702]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1238:14)
Oct 08 04:24:59 xoa xo-server[152702]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 08 04:24:59 xoa xo-server[152702]: code: 'VDI_IN_USE',
Oct 08 04:24:59 xoa xo-server[152702]: params: [ 'OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e', 'destroy' ],
Oct 08 04:24:59 xoa xo-server[152702]: call: undefined,
Oct 08 04:24:59 xoa xo-server[152702]: url: undefined,
Oct 08 04:24:59 xoa xo-server[152702]: task: task {
Oct 08 04:24:59 xoa xo-server[152702]: uuid: '240ae893-9d16-1762-572f-ec8da918dc49',
Oct 08 04:24:59 xoa xo-server[152702]: name_label: 'Async.VDI.destroy',
Oct 08 04:24:59 xoa xo-server[152702]: name_description: '',
Oct 08 04:24:59 xoa xo-server[152702]: allowed_operations: [],
Oct 08 04:24:59 xoa xo-server[152702]: current_operations: {},
Oct 08 04:24:59 xoa xo-server[152702]: created: '20241008T09:24:58Z',
Oct 08 04:24:59 xoa xo-server[152702]: finished: '20241008T09:24:59Z',
Oct 08 04:24:59 xoa xo-server[152702]: status: 'failure',
Oct 08 04:24:59 xoa xo-server[152702]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 08 04:24:59 xoa xo-server[152702]: progress: 1,
Oct 08 04:24:59 xoa xo-server[152702]: type: '<none/>',
Oct 08 04:24:59 xoa xo-server[152702]: result: '',
Oct 08 04:24:59 xoa xo-server[152702]: error_info: [Array],
Oct 08 04:24:59 xoa xo-server[152702]: other_config: {},
Oct 08 04:24:59 xoa xo-server[152702]: subtask_of: 'OpaqueRef:NULL',
Oct 08 04:24:59 xoa xo-server[152702]: subtasks: [],
Oct 08 04:24:59 xoa xo-server[152702]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))'
Oct 08 04:24:59 xoa xo-server[152702]: }
Oct 08 04:24:59 xoa xo-server[152702]: },
Oct 08 04:24:59 xoa xo-server[152702]: fn: 'destroy',
Oct 08 04:24:59 xoa xo-server[152702]: arguments: [ 'OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e' ],
Oct 08 04:24:59 xoa xo-server[152702]: pool: {
Oct 08 04:24:59 xoa xo-server[152702]: uuid: 'fe688bb2-b9ac-db7b-737a-cc457195f095',
Oct 08 04:24:59 xoa xo-server[152702]: name_label: 'Private Pool'
Oct 08 04:24:59 xoa xo-server[152702]: }
Oct 08 04:24:59 xoa xo-server[152702]: }
.....
Oct 08 04:30:12 xoa xo-server[152702]: 2024-10-08T09:30:12.650Z xo:xapi:vdi WARN Couldn't disconnect OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e from dom0 {
Oct 08 04:30:12 xoa xo-server[152702]: vdiRef: 'OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e',
Oct 08 04:30:12 xoa xo-server[152702]: vbdRef: 'OpaqueRef:872dd8e6-031c-4f12-adb5-c7bb61f9dc0f',
Oct 08 04:30:12 xoa xo-server[152702]: err: XapiError: OPERATION_NOT_ALLOWED(VBD '8817456d-774e-d82f-e63a-32a393491a0e' still attached to '9eebab44-b26e-4b6a-96e5-54734c459929')
Oct 08 04:30:12 xoa xo-server[152702]: at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Oct 08 04:30:12 xoa xo-server[152702]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
Oct 08 04:30:12 xoa xo-server[152702]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 08 04:30:12 xoa xo-server[152702]: code: 'OPERATION_NOT_ALLOWED',
Oct 08 04:30:12 xoa xo-server[152702]: params: [
Oct 08 04:30:12 xoa xo-server[152702]: "VBD '8817456d-774e-d82f-e63a-32a393491a0e' still attached to '9eebab44-b26e-4b6a-96e5-54734c459929'"
Oct 08 04:30:12 xoa xo-server[152702]: ],
Oct 08 04:30:12 xoa xo-server[152702]: call: { method: 'VBD.destroy', params: [Array] },
Oct 08 04:30:12 xoa xo-server[152702]: url: undefined,
Oct 08 04:30:12 xoa xo-server[152702]: task: undefined
Oct 08 04:30:12 xoa xo-server[152702]: }
Oct 08 04:30:12 xoa xo-server[152702]: }
Oct 08 04:30:17 xoa xo-server[152702]: 2024-10-08T09:30:17.708Z xo:xapi:vm WARN VM_destroy: failed to destroy VDI {
Oct 08 04:30:17 xoa xo-server[152702]: error: XapiError: VDI_IN_USE(OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e, destroy)
Oct 08 04:30:17 xoa xo-server[152702]: at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Oct 08 04:30:17 xoa xo-server[152702]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
Oct 08 04:30:17 xoa xo-server[152702]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1041:24)
Oct 08 04:30:17 xoa xo-server[152702]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1075:14
Oct 08 04:30:17 xoa xo-server[152702]: at Array.forEach (<anonymous>)
Oct 08 04:30:17 xoa xo-server[152702]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1065:12)
Oct 08 04:30:17 xoa xo-server[152702]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1238:14)
Oct 08 04:30:17 xoa xo-server[152702]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 08 04:30:17 xoa xo-server[152702]: code: 'VDI_IN_USE',
Oct 08 04:30:17 xoa xo-server[152702]: params: [ 'OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e', 'destroy' ],
Oct 08 04:30:17 xoa xo-server[152702]: call: undefined,
Oct 08 04:30:17 xoa xo-server[152702]: url: undefined,
Oct 08 04:30:17 xoa xo-server[152702]: task: task {
Oct 08 04:30:17 xoa xo-server[152702]: uuid: '81ce7cd4-6e11-e60d-6fc2-22522ba2dd74',
Oct 08 04:30:17 xoa xo-server[152702]: name_label: 'Async.VDI.destroy',
Oct 08 04:30:17 xoa xo-server[152702]: name_description: '',
Oct 08 04:30:17 xoa xo-server[152702]: allowed_operations: [],
Oct 08 04:30:17 xoa xo-server[152702]: current_operations: {},
Oct 08 04:30:17 xoa xo-server[152702]: created: '20241008T09:30:17Z',
Oct 08 04:30:17 xoa xo-server[152702]: finished: '20241008T09:30:17Z',
Oct 08 04:30:17 xoa xo-server[152702]: status: 'failure',
Oct 08 04:30:17 xoa xo-server[152702]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 08 04:30:17 xoa xo-server[152702]: progress: 1,
Oct 08 04:30:17 xoa xo-server[152702]: type: '<none/>',
Oct 08 04:30:17 xoa xo-server[152702]: result: '',
Oct 08 04:30:17 xoa xo-server[152702]: error_info: [Array],
Oct 08 04:30:17 xoa xo-server[152702]: other_config: {},
Oct 08 04:30:17 xoa xo-server[152702]: subtask_of: 'OpaqueRef:NULL',
Oct 08 04:30:17 xoa xo-server[152702]: subtasks: [],
Oct 08 04:30:17 xoa xo-server[152702]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))'
Oct 08 04:30:17 xoa xo-server[152702]: }
Oct 08 04:30:17 xoa xo-server[152702]: },
Oct 08 04:30:17 xoa xo-server[152702]: vdiRef: 'OpaqueRef:7d08bb03-3247-4ace-98b0-4110851a893e',
Oct 08 04:30:17 xoa xo-server[152702]: vmRef: 'OpaqueRef:fac41617-6192-46b5-9d01-993da8ce039f'
Oct 08 04:30:17 xoa xo-server[152702]: }
Oct 08 04:30:18 xoa xo-server[152702]: 2024-10-08T09:30:18.041Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
Oct 08 04:30:18 xoa xo-server[152702]: path: '/xo-vm-backups/d5d0334c-a7e3-b29f-51ca-1be9c211d2c1/vdis/05ce27a6-b447-4b12-b963-7e8d4978c95a/9f8da2fd-a08d-43c2-a5b1-ce6125cf52f5/20241007T135637Z.vhd'
Oct 08 04:30:18 xoa xo-server[152702]: }
Oct 08 04:30:18 xoa xo-server[152702]: 2024-10-08T09:30:18.041Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
Oct 08 04:30:18 xoa xo-server[152702]: path: '/xo-vm-backups/d5d0334c-a7e3-b29f-51ca-1be9c211d2c1/vdis/05ce27a6-b447-4b12-b963-7e8d4978c95a/c6853f48-4b06-4c34-9707-b68f9054e6fc/20241007T135637Z.vhd'
Oct 08 04:30:18 xoa xo-server[152702]: }
Oct 08 04:30:18 xoa xo-server[152702]: 2024-10-08T09:30:18.148Z xo:backups:worker INFO backup has ended
Oct 08 04:30:48 xoa xo-server[152702]: 2024-10-08T09:30:48.256Z xo:backups:worker WARN worker process did not exit automatically, forcing...
Oct 08 04:30:48 xoa xo-server[152702]: 2024-10-08T09:30:48.257Z xo:backups:worker INFO process will exit {
Oct 08 04:30:48 xoa xo-server[152702]: duration: 1107117425,
Oct 08 04:30:48 xoa xo-server[152702]: exitCode: 0,
Oct 08 04:30:48 xoa xo-server[152702]: resourceUsage: {
Oct 08 04:30:48 xoa xo-server[152702]: userCPUTime: 839002829,
Oct 08 04:30:48 xoa xo-server[152702]: systemCPUTime: 176610166,
Oct 08 04:30:48 xoa xo-server[152702]: maxRSS: 89720,
Oct 08 04:30:48 xoa xo-server[152702]: sharedMemorySize: 0,
Oct 08 04:30:48 xoa xo-server[152702]: unsharedDataSize: 0,
Oct 08 04:30:48 xoa xo-server[152702]: unsharedStackSize: 0,
Oct 08 04:30:48 xoa xo-server[152702]: minorPageFault: 4860813,
Oct 08 04:30:48 xoa xo-server[152702]: majorPageFault: 0,
Oct 08 04:30:48 xoa xo-server[152702]: swappedOut: 0,
Oct 08 04:30:48 xoa xo-server[152702]: fsRead: 4224,
Oct 08 04:30:48 xoa xo-server[152702]: fsWrite: 209821368,
Oct 08 04:30:48 xoa xo-server[152702]: ipcSent: 0,
Oct 08 04:30:48 xoa xo-server[152702]: ipcReceived: 0,
Oct 08 04:30:48 xoa xo-server[152702]: signalsCount: 0,
Oct 08 04:30:48 xoa xo-server[152702]: voluntaryContextSwitches: 451688,
Oct 08 04:30:48 xoa xo-server[152702]: involuntaryContextSwitches: 480143
Oct 08 04:30:48 xoa xo-server[152702]: },
Oct 08 04:30:48 xoa xo-server[152702]: summary: { duration: '18m', cpuUsage: '92%', memoryUsage: '87.62 MiB' }
Oct 08 04:30:48 xoa xo-server[152702]: }
@olivierlambert
Same behaviour on XOA, Full backup OK, delta succeedes, but 2nd disk stuck on control domain:
Current version: 5.98.1 - XOA build: 20241004
PS. just noticed that while the delta is successfully ran, the backuplog say:
Retry the VM backup due to an error
With message:
"The value of \"offset\" is out of range. It must be >= 0 and <= 102399. Received 102400"
2nd attempt ran successfully, while leaving the vdi orphaned, on both XO and XOA.
Oct 07 08:44:39 xoa xo-server[4796]: 2024-10-07T13:44:39.083Z xo:backups:worker INFO starting backup
Oct 07 08:54:50 xoa xo-server[4796]: 2024-10-07T13:54:50.199Z xo:xapi:vdi INFO OpaqueRef:841ceab7-91b7-48a4-a85c-f4505c1614bd has been disconnected from dom0 {
Oct 07 08:54:50 xoa xo-server[4796]: vdiRef: 'OpaqueRef:01c25ee1-a119-4f1d-bfdc-f294a83fee0b',
Oct 07 08:54:50 xoa xo-server[4796]: vbdRef: 'OpaqueRef:841ceab7-91b7-48a4-a85c-f4505c1614bd'
Oct 07 08:54:50 xoa xo-server[4796]: }
Oct 07 08:54:58 xoa xo-server[4796]: 2024-10-07T13:54:58.283Z xo:backups:worker INFO backup has ended
Oct 07 08:55:09 xoa xo-server[4796]: 2024-10-07T13:55:09.682Z xo:backups:worker INFO process will exit {
Oct 07 08:55:09 xoa xo-server[4796]: duration: 630597722,
Oct 07 08:55:09 xoa xo-server[4796]: exitCode: 0,
Oct 07 08:55:09 xoa xo-server[4796]: resourceUsage: {
Oct 07 08:55:09 xoa xo-server[4796]: userCPUTime: 709081409,
Oct 07 08:55:09 xoa xo-server[4796]: systemCPUTime: 152978165,
Oct 07 08:55:09 xoa xo-server[4796]: maxRSS: 99824,
Oct 07 08:55:09 xoa xo-server[4796]: sharedMemorySize: 0,
Oct 07 08:55:09 xoa xo-server[4796]: unsharedDataSize: 0,
Oct 07 08:55:09 xoa xo-server[4796]: unsharedStackSize: 0,
Oct 07 08:55:09 xoa xo-server[4796]: minorPageFault: 1526509,
Oct 07 08:55:09 xoa xo-server[4796]: majorPageFault: 1,
Oct 07 08:55:09 xoa xo-server[4796]: swappedOut: 0,
Oct 07 08:55:09 xoa xo-server[4796]: fsRead: 4736,
Oct 07 08:55:09 xoa xo-server[4796]: fsWrite: 209788416,
Oct 07 08:55:09 xoa xo-server[4796]: ipcSent: 0,
Oct 07 08:55:09 xoa xo-server[4796]: ipcReceived: 0,
Oct 07 08:55:09 xoa xo-server[4796]: signalsCount: 0,
Oct 07 08:55:09 xoa xo-server[4796]: voluntaryContextSwitches: 397016,
Oct 07 08:55:09 xoa xo-server[4796]: involuntaryContextSwitches: 441949
Oct 07 08:55:09 xoa xo-server[4796]: },
Oct 07 08:55:09 xoa xo-server[4796]: summary: { duration: '11m', cpuUsage: '137%', memoryUsage: '97.48 MiB' }
Oct 07 08:55:09 xoa xo-server[4796]: }
Oct 07 08:55:47 xoa xo-server[6339]: 2024-10-07T13:55:47.446Z xo:backups:worker INFO starting backup
Oct 07 08:55:53 xoa xo-server[6339]: 2024-10-07T13:55:53.223Z xo:xapi:vdi INFO found changed blocks {
Oct 07 08:55:53 xoa xo-server[6339]: changedBlocks: <Buffer 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 511950 more bytes>
Oct 07 08:55:53 xoa xo-server[6339]: }
Oct 07 08:56:11 xoa xo-server[6339]: 2024-10-07T13:56:11.907Z xo:xapi:vdi INFO found changed blocks {
Oct 07 08:56:11 xoa xo-server[6339]: changedBlocks: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 102353 more bytes>
Oct 07 08:56:11 xoa xo-server[6339]: }
Oct 07 08:56:14 xoa xo-server[6339]: 2024-10-07T13:56:14.369Z xo:xapi:vdi INFO OpaqueRef:18023c91-93c8-489b-85e4-f2bda338258c has been disconnected from dom0 {
Oct 07 08:56:14 xoa xo-server[6339]: vdiRef: 'OpaqueRef:b438dd7b-d27c-4184-99e5-fddfb44901b0',
Oct 07 08:56:14 xoa xo-server[6339]: vbdRef: 'OpaqueRef:18023c91-93c8-489b-85e4-f2bda338258c'
Oct 07 08:56:14 xoa xo-server[6339]: }
Oct 07 09:07:42 xoa xo-server[6339]: 2024-10-07T14:07:42.981Z xo:xapi WARN retry {
Oct 07 09:07:42 xoa xo-server[6339]: attemptNumber: 0,
Oct 07 09:07:42 xoa xo-server[6339]: delay: 5000,
Oct 07 09:07:42 xoa xo-server[6339]: error: XapiError: VDI_IN_USE(OpaqueRef:69e0e65f-94ee-468a-b919-2b22d3db670a, destroy)
Oct 07 09:07:42 xoa xo-server[6339]: at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Oct 07 09:07:42 xoa xo-server[6339]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
Oct 07 09:07:42 xoa xo-server[6339]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1076:24)
Oct 07 09:07:42 xoa xo-server[6339]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1110:14
Oct 07 09:07:42 xoa xo-server[6339]: at Array.forEach (<anonymous>)
Oct 07 09:07:42 xoa xo-server[6339]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1100:12)
Oct 07 09:07:42 xoa xo-server[6339]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1273:14)
Oct 07 09:07:42 xoa xo-server[6339]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 07 09:07:42 xoa xo-server[6339]: code: 'VDI_IN_USE',
Oct 07 09:07:42 xoa xo-server[6339]: params: [ 'OpaqueRef:69e0e65f-94ee-468a-b919-2b22d3db670a', 'destroy' ],
Oct 07 09:07:42 xoa xo-server[6339]: call: undefined,
Oct 07 09:07:42 xoa xo-server[6339]: url: undefined,
Oct 07 09:07:42 xoa xo-server[6339]: task: task {
Oct 07 09:07:42 xoa xo-server[6339]: uuid: '30d84ccc-fe22-fd16-6697-4615df096ee6',
Oct 07 09:07:42 xoa xo-server[6339]: name_label: 'Async.VDI.destroy',
Oct 07 09:07:42 xoa xo-server[6339]: name_description: '',
Oct 07 09:07:42 xoa xo-server[6339]: allowed_operations: [],
Oct 07 09:07:42 xoa xo-server[6339]: current_operations: {},
Oct 07 09:07:42 xoa xo-server[6339]: created: '20241007T14:07:33Z',
Oct 07 09:07:42 xoa xo-server[6339]: finished: '20241007T14:07:42Z',
Oct 07 09:07:42 xoa xo-server[6339]: status: 'failure',
Oct 07 09:07:42 xoa xo-server[6339]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 07 09:07:42 xoa xo-server[6339]: progress: 1,
Oct 07 09:07:42 xoa xo-server[6339]: type: '<none/>',
Oct 07 09:07:42 xoa xo-server[6339]: result: '',
Oct 07 09:07:42 xoa xo-server[6339]: error_info: [Array],
Oct 07 09:07:42 xoa xo-server[6339]: other_config: {},
Oct 07 09:07:42 xoa xo-server[6339]: subtask_of: 'OpaqueRef:NULL',
Oct 07 09:07:42 xoa xo-server[6339]: subtasks: [],
Oct 07 09:07:42 xoa xo-server[6339]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename o>
Oct 07 09:07:42 xoa xo-server[6339]: }
Oct 07 09:07:42 xoa xo-server[6339]: },
Oct 07 09:07:42 xoa xo-server[6339]: fn: 'destroy',
Oct 07 09:07:42 xoa xo-server[6339]: arguments: [ 'OpaqueRef:69e0e65f-94ee-468a-b919-2b22d3db670a' ],
Oct 07 09:07:42 xoa xo-server[6339]: pool: {
Oct 07 09:07:42 xoa xo-server[6339]: uuid: 'fe688bb2-b9ac-db7b-737a-cc457195f095',
Oct 07 09:07:42 xoa xo-server[6339]: name_label: 'SGO Pool'
Oct 07 09:07:42 xoa xo-server[6339]: }
Oct 07 09:07:42 xoa xo-server[6339]: }
Oct 07 09:08:28 xoa xo-server[6339]: 2024-10-07T14:08:28.493Z xo:xapi:vm WARN VM_destroy: failed to destroy VDI {
Oct 07 09:08:28 xoa xo-server[6339]: error: XapiError: VDI_IN_USE(OpaqueRef:69e0e65f-94ee-468a-b919-2b22d3db670a, destroy)
Oct 07 09:08:28 xoa xo-server[6339]: at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Oct 07 09:08:28 xoa xo-server[6339]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
Oct 07 09:08:28 xoa xo-server[6339]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1076:24)
Oct 07 09:08:28 xoa xo-server[6339]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1110:14
Oct 07 09:08:28 xoa xo-server[6339]: at Array.forEach (<anonymous>)
Oct 07 09:08:28 xoa xo-server[6339]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1100:12)
Oct 07 09:08:28 xoa xo-server[6339]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1273:14)
Oct 07 09:08:28 xoa xo-server[6339]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 07 09:08:28 xoa xo-server[6339]: code: 'VDI_IN_USE',
Oct 07 09:08:28 xoa xo-server[6339]: params: [ 'OpaqueRef:69e0e65f-94ee-468a-b919-2b22d3db670a', 'destroy' ],
Oct 07 09:08:28 xoa xo-server[6339]: call: undefined,
Oct 07 09:08:28 xoa xo-server[6339]: url: undefined,
Oct 07 09:08:28 xoa xo-server[6339]: task: task {
Oct 07 09:08:28 xoa xo-server[6339]: uuid: '76ceadda-070d-3e2e-282b-76c58abd13b1',
Oct 07 09:08:28 xoa xo-server[6339]: name_label: 'Async.VDI.destroy',
Oct 07 09:08:28 xoa xo-server[6339]: name_description: '',
Oct 07 09:08:28 xoa xo-server[6339]: allowed_operations: [],
Oct 07 09:08:28 xoa xo-server[6339]: current_operations: {},
Oct 07 09:08:28 xoa xo-server[6339]: created: '20241007T14:08:28Z',
Oct 07 09:08:28 xoa xo-server[6339]: finished: '20241007T14:08:28Z',
Oct 07 09:08:28 xoa xo-server[6339]: status: 'failure',
Oct 07 09:08:28 xoa xo-server[6339]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 07 09:08:28 xoa xo-server[6339]: progress: 1,
Oct 07 09:08:28 xoa xo-server[6339]: type: '<none/>',
Oct 07 09:08:28 xoa xo-server[6339]: result: '',
Oct 07 09:08:28 xoa xo-server[6339]: error_info: [Array],
Oct 07 09:08:28 xoa xo-server[6339]: other_config: {},
Oct 07 09:08:28 xoa xo-server[6339]: subtask_of: 'OpaqueRef:NULL',
Oct 07 09:08:28 xoa xo-server[6339]: subtasks: [],
Oct 07 09:08:28 xoa xo-server[6339]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename o>
Oct 07 09:08:28 xoa xo-server[6339]: }
Oct 07 09:08:28 xoa xo-server[6339]: },
Oct 07 09:08:28 xoa xo-server[6339]: vdiRef: 'OpaqueRef:69e0e65f-94ee-468a-b919-2b22d3db670a',
Oct 07 09:08:28 xoa xo-server[6339]: vmRef: 'OpaqueRef:c246a561-2409-4e72-900c-b2d259707e22'
Oct 07 09:08:28 xoa xo-server[6339]: }
Oct 07 09:08:28 xoa xo-server[6339]: 2024-10-07T14:08:28.677Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
Oct 07 09:08:28 xoa xo-server[6339]: path: '/xo-vm-backups/d5d0334c-a7e3-b29f-51ca-1be9c211d2c1/vdis/05ce27a6-b447-4b12-b963-7e8d4978c95a/c6853f48-4b06-4c34-9707-b68f9054e6fc/20241007T134447Z.vhd'
Oct 07 09:08:28 xoa xo-server[6339]: }
Oct 07 09:08:28 xoa xo-server[6339]: 2024-10-07T14:08:28.678Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
Oct 07 09:08:28 xoa xo-server[6339]: path: '/xo-vm-backups/d5d0334c-a7e3-b29f-51ca-1be9c211d2c1/vdis/05ce27a6-b447-4b12-b963-7e8d4978c95a/9f8da2fd-a08d-43c2-a5b1-ce6125cf52f5/20241007T134447Z.vhd'
Oct 07 09:08:28 xoa xo-server[6339]: }
Oct 07 09:08:28 xoa xo-server[6339]: 2024-10-07T14:08:28.788Z xo:backups:worker INFO backup has ended
Hi,
I have a problem with a single VM running Almalinux 9. The VM has 2 disks, 50GB and 250GB disk.
Every time i do Delta backup the backup does succeed, but the larger disk is ALWAYS stuck on Control Domain and i have to manually "forget" it, or next backup will fail, which is annoying.
This does not occur on full backup, only delta.
I removed all previous snapshots and backup and retried and same thing is happening.
Using XO with latest patches: commit a5967.
This issue started occurring straight after the CBT feature was added.
Using NBD = 1
Here is XO LOG for the backup:
Omitted attempts 2-7.... since they don't fit here
Oct 7 11:33:43 xo-ce xo-server[11768]: 2024-10-07T11:33:43.770Z xo:backups:worker INFO starting backup
Oct 7 11:33:49 xo-ce xo-server[11768]: 2024-10-07T11:33:49.717Z xo:xapi:vdi INFO found changed blocks {
Oct 7 11:33:49 xo-ce xo-server[11768]: changedBlocks: <Buffer 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 511950 more bytes>
Oct 7 11:33:49 xo-ce xo-server[11768]: }
Oct 7 11:34:09 xo-ce xo-server[11768]: 2024-10-07T11:34:09.098Z xo:xapi:vdi INFO found changed blocks {
Oct 7 11:34:09 xo-ce xo-server[11768]: changedBlocks: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... 102353 more bytes>
Oct 7 11:34:09 xo-ce xo-server[11768]: }
Oct 7 11:34:11 xo-ce xo-server[11768]: 2024-10-07T11:34:11.070Z xo:xapi:vdi INFO OpaqueRef:c1f4b3cf-996d-45ba-80ae-59ef7c9cb8ef has been disconnected from dom0 {
Oct 7 11:34:11 xo-ce xo-server[11768]: vdiRef: 'OpaqueRef:2fd1d459-46b5-473a-9549-a3e0b401209f',
Oct 7 11:34:11 xo-ce xo-server[11768]: vbdRef: 'OpaqueRef:c1f4b3cf-996d-45ba-80ae-59ef7c9cb8ef'
Oct 7 11:34:11 xo-ce xo-server[11768]: }
Oct 7 11:40:47 xo-ce xo-server[11768]: 2024-10-07T11:40:47.534Z xo:xapi:vdi INFO OpaqueRef:7e94c0da-be46-45f8-9a6b-7e9eb04928b1 has been disconnected from dom0 {
Oct 7 11:40:47 xo-ce xo-server[11768]: vdiRef: 'OpaqueRef:f8be3361-0c5c-48b4-bf51-baee4d02cb20',
Oct 7 11:40:47 xo-ce xo-server[11768]: vbdRef: 'OpaqueRef:7e94c0da-be46-45f8-9a6b-7e9eb04928b1'
Oct 7 11:40:47 xo-ce xo-server[11768]: }
Oct 7 11:41:59 xo-ce xo-server[11768]: 2024-10-07T11:41:59.303Z xo:xapi:vdi INFO OpaqueRef:2e1231be-f65b-4fa4-bb8d-3871ddabfa87 has been disconnected from dom0 {
Oct 7 11:41:59 xo-ce xo-server[11768]: vdiRef: 'OpaqueRef:cb2af589-ae7e-4b5c-b657-0c2636b3677f',
Oct 7 11:41:59 xo-ce xo-server[11768]: vbdRef: 'OpaqueRef:2e1231be-f65b-4fa4-bb8d-3871ddabfa87'
Oct 7 11:41:59 xo-ce xo-server[11768]: }
Oct 7 11:42:11 xo-ce xo-server[11768]: 2024-10-07T11:42:11.742Z xo:xapi WARN retry {
Oct 7 11:42:11 xo-ce xo-server[11768]: attemptNumber: 0,
Oct 7 11:42:11 xo-ce xo-server[11768]: delay: 5000,
Oct 7 11:42:11 xo-ce xo-server[11768]: error: XapiError: VDI_IN_USE(OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92, destroy)
Oct 7 11:42:11 xo-ce xo-server[11768]: at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_XapiError.mjs:16:12)
Oct 7 11:42:11 xo-ce xo-server[11768]: at default (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_getTaskResult.mjs:13:29)
Oct 7 11:42:11 xo-ce xo-server[11768]: at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1041:24)
Oct 7 11:42:11 xo-ce xo-server[11768]: at file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1075:14
Oct 7 11:42:11 xo-ce xo-server[11768]: at Array.forEach (<anonymous>)
Oct 7 11:42:11 xo-ce xo-server[11768]: at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1065:12)
Oct 7 11:42:11 xo-ce xo-server[11768]: at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1238:14)
Oct 7 11:42:11 xo-ce xo-server[11768]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 7 11:42:11 xo-ce xo-server[11768]: code: 'VDI_IN_USE',
Oct 7 11:42:11 xo-ce xo-server[11768]: params: [ 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92', 'destroy' ],
Oct 7 11:42:11 xo-ce xo-server[11768]: call: undefined,
Oct 7 11:42:11 xo-ce xo-server[11768]: url: undefined,
Oct 7 11:42:11 xo-ce xo-server[11768]: task: task {
Oct 7 11:42:11 xo-ce xo-server[11768]: uuid: '7bb5adaa-5b49-2227-3ede-759420836f59',
Oct 7 11:42:11 xo-ce xo-server[11768]: name_label: 'Async.VDI.destroy',
Oct 7 11:42:11 xo-ce xo-server[11768]: name_description: '',
Oct 7 11:42:11 xo-ce xo-server[11768]: allowed_operations: [],
Oct 7 11:42:11 xo-ce xo-server[11768]: current_operations: {},
Oct 7 11:42:11 xo-ce xo-server[11768]: created: '20241007T11:41:59Z',
Oct 7 11:42:11 xo-ce xo-server[11768]: finished: '20241007T11:42:11Z',
Oct 7 11:42:11 xo-ce xo-server[11768]: status: 'failure',
Oct 7 11:42:11 xo-ce xo-server[11768]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 7 11:42:11 xo-ce xo-server[11768]: progress: 1,
Oct 7 11:42:11 xo-ce xo-server[11768]: type: '<none/>',
Oct 7 11:42:11 xo-ce xo-server[11768]: result: '',
Oct 7 11:42:11 xo-ce xo-server[11768]: error_info: [Array],
Oct 7 11:42:11 xo-ce xo-server[11768]: other_config: {},
Oct 7 11:42:11 xo-ce xo-server[11768]: subtask_of: 'OpaqueRef:NULL',
Oct 7 11:42:11 xo-ce xo-server[11768]: subtasks: [],
Oct 7 11:42:11 xo-ce xo-server[11768]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))'
Oct 7 11:42:11 xo-ce xo-server[11768]: }
Oct 7 11:42:11 xo-ce xo-server[11768]: },
Oct 7 11:42:11 xo-ce xo-server[11768]: fn: 'destroy',
Oct 7 11:42:11 xo-ce xo-server[11768]: arguments: [ 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92' ],
Oct 7 11:42:11 xo-ce xo-server[11768]: pool: {
Oct 7 11:42:11 xo-ce xo-server[11768]: uuid: 'fe688bb2-b9ac-db7b-737a-cc457195f095',
Oct 7 11:42:11 xo-ce xo-server[11768]: name_label: 'SGO Pool'
Oct 7 11:42:11 xo-ce xo-server[11768]: }
Oct 7 11:42:11 xo-ce xo-server[11768]: }
Oct 7 11:42:42 xo-ce xo-server[11768]: 2024-10-07T11:42:42.061Z xo:xapi:vdi WARN Couldn't disconnect OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92 from dom0 {
Oct 7 11:42:42 xo-ce xo-server[11768]: vdiRef: 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92',
Oct 7 11:42:42 xo-ce xo-server[11768]: vbdRef: 'OpaqueRef:2b0c22e7-2794-471a-bd85-145cf302f8e4',
Oct 7 11:42:42 xo-ce xo-server[11768]: err: XapiError: OPERATION_NOT_ALLOWED(VBD '0da0be96-45ef-684e-b956-7b5463cd9b09' still attached to 'f3fd6bd3-a622-4838-a7b6-e6e5e2cc6d04')
Oct 7 11:42:42 xo-ce xo-server[11768]: at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_XapiError.mjs:16:12)
Oct 7 11:42:42 xo-ce xo-server[11768]: at file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/transports/json-rpc.mjs:38:21
Oct 7 11:42:42 xo-ce xo-server[11768]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 7 11:42:42 xo-ce xo-server[11768]: code: 'OPERATION_NOT_ALLOWED',
Oct 7 11:42:42 xo-ce xo-server[11768]: params: [
Oct 7 11:42:42 xo-ce xo-server[11768]: "VBD '0da0be96-45ef-684e-b956-7b5463cd9b09' still attached to 'f3fd6bd3-a622-4838-a7b6-e6e5e2cc6d04'"
Oct 7 11:42:42 xo-ce xo-server[11768]: ],
Oct 7 11:42:42 xo-ce xo-server[11768]: call: { method: 'VBD.destroy', params: [Array] },
Oct 7 11:42:42 xo-ce xo-server[11768]: url: undefined,
Oct 7 11:42:42 xo-ce xo-server[11768]: task: undefined
Oct 7 11:42:42 xo-ce xo-server[11768]: }
Oct 7 11:42:42 xo-ce xo-server[11768]: }
Oct 7 11:46:54 xo-ce xo-server[11768]: 2024-10-07T11:46:54.842Z xo:xapi WARN retry {
Oct 7 11:46:54 xo-ce xo-server[11768]: attemptNumber: 8,
Oct 7 11:46:54 xo-ce xo-server[11768]: delay: 5000,
Oct 7 11:46:54 xo-ce xo-server[11768]: error: XapiError: VDI_IN_USE(OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92, destroy)
Oct 7 11:46:54 xo-ce xo-server[11768]: at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_XapiError.mjs:16:12)
Oct 7 11:46:54 xo-ce xo-server[11768]: at default (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_getTaskResult.mjs:13:29)
Oct 7 11:46:54 xo-ce xo-server[11768]: at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1041:24)
Oct 7 11:46:54 xo-ce xo-server[11768]: at file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1075:14
Oct 7 11:46:54 xo-ce xo-server[11768]: at Array.forEach (<anonymous>)
Oct 7 11:46:54 xo-ce xo-server[11768]: at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1065:12)
Oct 7 11:46:54 xo-ce xo-server[11768]: at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1238:14)
Oct 7 11:46:54 xo-ce xo-server[11768]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 7 11:46:54 xo-ce xo-server[11768]: code: 'VDI_IN_USE',
Oct 7 11:46:54 xo-ce xo-server[11768]: params: [ 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92', 'destroy' ],
Oct 7 11:46:54 xo-ce xo-server[11768]: call: undefined,
Oct 7 11:46:54 xo-ce xo-server[11768]: url: undefined,
Oct 7 11:46:54 xo-ce xo-server[11768]: task: task {
Oct 7 11:46:54 xo-ce xo-server[11768]: uuid: '70afe347-815d-806b-0eed-dd6239adcf0e',
Oct 7 11:46:54 xo-ce xo-server[11768]: name_label: 'Async.VDI.destroy',
Oct 7 11:46:54 xo-ce xo-server[11768]: name_description: '',
Oct 7 11:46:54 xo-ce xo-server[11768]: allowed_operations: [],
Oct 7 11:46:54 xo-ce xo-server[11768]: current_operations: {},
Oct 7 11:46:54 xo-ce xo-server[11768]: created: '20241007T11:46:54Z',
Oct 7 11:46:54 xo-ce xo-server[11768]: finished: '20241007T11:46:54Z',
Oct 7 11:46:54 xo-ce xo-server[11768]: status: 'failure',
Oct 7 11:46:54 xo-ce xo-server[11768]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 7 11:46:54 xo-ce xo-server[11768]: progress: 1,
Oct 7 11:46:54 xo-ce xo-server[11768]: type: '<none/>',
Oct 7 11:46:54 xo-ce xo-server[11768]: result: '',
Oct 7 11:46:54 xo-ce xo-server[11768]: error_info: [Array],
Oct 7 11:46:54 xo-ce xo-server[11768]: other_config: {},
Oct 7 11:46:54 xo-ce xo-server[11768]: subtask_of: 'OpaqueRef:NULL',
Oct 7 11:46:54 xo-ce xo-server[11768]: subtasks: [],
Oct 7 11:46:54 xo-ce xo-server[11768]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))'
Oct 7 11:46:54 xo-ce xo-server[11768]: }
Oct 7 11:46:54 xo-ce xo-server[11768]: },
Oct 7 11:46:54 xo-ce xo-server[11768]: fn: 'destroy',
Oct 7 11:46:54 xo-ce xo-server[11768]: arguments: [ 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92' ],
Oct 7 11:46:54 xo-ce xo-server[11768]: pool: {
Oct 7 11:46:54 xo-ce xo-server[11768]: uuid: 'fe688bb2-b9ac-db7b-737a-cc457195f095',
Oct 7 11:46:54 xo-ce xo-server[11768]: name_label: 'SGO Pool'
Oct 7 11:46:54 xo-ce xo-server[11768]: }
Oct 7 11:46:54 xo-ce xo-server[11768]: }
Oct 7 11:47:25 xo-ce xo-server[11768]: 2024-10-07T11:47:25.194Z xo:xapi:vdi WARN Couldn't disconnect OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92 from dom0 {
Oct 7 11:47:25 xo-ce xo-server[11768]: vdiRef: 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92',
Oct 7 11:47:25 xo-ce xo-server[11768]: vbdRef: 'OpaqueRef:2b0c22e7-2794-471a-bd85-145cf302f8e4',
Oct 7 11:47:25 xo-ce xo-server[11768]: err: XapiError: OPERATION_NOT_ALLOWED(VBD '0da0be96-45ef-684e-b956-7b5463cd9b09' still attached to 'f3fd6bd3-a622-4838-a7b6-e6e5e2cc6d04')
Oct 7 11:47:25 xo-ce xo-server[11768]: at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_XapiError.mjs:16:12)
Oct 7 11:47:25 xo-ce xo-server[11768]: at file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/transports/json-rpc.mjs:38:21
Oct 7 11:47:25 xo-ce xo-server[11768]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 7 11:47:25 xo-ce xo-server[11768]: code: 'OPERATION_NOT_ALLOWED',
Oct 7 11:47:25 xo-ce xo-server[11768]: params: [
Oct 7 11:47:25 xo-ce xo-server[11768]: "VBD '0da0be96-45ef-684e-b956-7b5463cd9b09' still attached to 'f3fd6bd3-a622-4838-a7b6-e6e5e2cc6d04'"
Oct 7 11:47:25 xo-ce xo-server[11768]: ],
Oct 7 11:47:25 xo-ce xo-server[11768]: call: { method: 'VBD.destroy', params: [Array] },
Oct 7 11:47:25 xo-ce xo-server[11768]: url: undefined,
Oct 7 11:47:25 xo-ce xo-server[11768]: task: undefined
Oct 7 11:47:25 xo-ce xo-server[11768]: }
Oct 7 11:47:25 xo-ce xo-server[11768]: }
Oct 7 11:47:30 xo-ce xo-server[11768]: 2024-10-07T11:47:30.323Z xo:xapi:vm WARN VM_destroy: failed to destroy VDI {
Oct 7 11:47:30 xo-ce xo-server[11768]: error: XapiError: VDI_IN_USE(OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92, destroy)
Oct 7 11:47:30 xo-ce xo-server[11768]: at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_XapiError.mjs:16:12)
Oct 7 11:47:30 xo-ce xo-server[11768]: at default (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/_getTaskResult.mjs:13:29)
Oct 7 11:47:30 xo-ce xo-server[11768]: at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1041:24)
Oct 7 11:47:30 xo-ce xo-server[11768]: at file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1075:14
Oct 7 11:47:30 xo-ce xo-server[11768]: at Array.forEach (<anonymous>)
Oct 7 11:47:30 xo-ce xo-server[11768]: at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1065:12)
Oct 7 11:47:30 xo-ce xo-server[11768]: at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410070635/packages/xen-api/index.mjs:1238:14)
Oct 7 11:47:30 xo-ce xo-server[11768]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Oct 7 11:47:30 xo-ce xo-server[11768]: code: 'VDI_IN_USE',
Oct 7 11:47:30 xo-ce xo-server[11768]: params: [ 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92', 'destroy' ],
Oct 7 11:47:30 xo-ce xo-server[11768]: call: undefined,
Oct 7 11:47:30 xo-ce xo-server[11768]: url: undefined,
Oct 7 11:47:30 xo-ce xo-server[11768]: task: task {
Oct 7 11:47:30 xo-ce xo-server[11768]: uuid: '5c688ec4-5523-c1f1-89dc-a881abdbaee1',
Oct 7 11:47:30 xo-ce xo-server[11768]: name_label: 'Async.VDI.destroy',
Oct 7 11:47:30 xo-ce xo-server[11768]: name_description: '',
Oct 7 11:47:30 xo-ce xo-server[11768]: allowed_operations: [],
Oct 7 11:47:30 xo-ce xo-server[11768]: current_operations: {},
Oct 7 11:47:30 xo-ce xo-server[11768]: created: '20241007T11:47:30Z',
Oct 7 11:47:30 xo-ce xo-server[11768]: finished: '20241007T11:47:30Z',
Oct 7 11:47:30 xo-ce xo-server[11768]: status: 'failure',
Oct 7 11:47:30 xo-ce xo-server[11768]: resident_on: 'OpaqueRef:010eebba-be27-489f-9f87-d06c8b675f19',
Oct 7 11:47:30 xo-ce xo-server[11768]: progress: 1,
Oct 7 11:47:30 xo-ce xo-server[11768]: type: '<none/>',
Oct 7 11:47:30 xo-ce xo-server[11768]: result: '',
Oct 7 11:47:30 xo-ce xo-server[11768]: error_info: [Array],
Oct 7 11:47:30 xo-ce xo-server[11768]: other_config: {},
Oct 7 11:47:30 xo-ce xo-server[11768]: subtask_of: 'OpaqueRef:NULL',
Oct 7 11:47:30 xo-ce xo-server[11768]: subtasks: [],
Oct 7 11:47:30 xo-ce xo-server[11768]: backtrace: '(((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 4711))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))'
Oct 7 11:47:30 xo-ce xo-server[11768]: }
Oct 7 11:47:30 xo-ce xo-server[11768]: },
Oct 7 11:47:30 xo-ce xo-server[11768]: vdiRef: 'OpaqueRef:8673586d-a9fc-4651-883a-ad333c13dc92',
Oct 7 11:47:30 xo-ce xo-server[11768]: vmRef: 'OpaqueRef:239e4aa1-b3ce-4fa0-abe4-3fb6803a990e'
Oct 7 11:47:30 xo-ce xo-server[11768]: }
Oct 7 11:47:30 xo-ce xo-server[11768]: 2024-10-07T11:47:30.474Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
Oct 7 11:47:30 xo-ce xo-server[11768]: path: '/xo-vm-backups/d5d0334c-a7e3-b29f-51ca-1be9c211d2c1/vdis/5a7a5647-d2ca-4014-b87b-06c621c09a20/9f8da2fd-a08d-43c2-a5b1-ce6125cf52f5/20241007T111850Z.vhd'
Oct 7 11:47:30 xo-ce xo-server[11768]: }
Oct 7 11:47:30 xo-ce xo-server[11768]: 2024-10-07T11:47:30.475Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
Oct 7 11:47:30 xo-ce xo-server[11768]: path: '/xo-vm-backups/d5d0334c-a7e3-b29f-51ca-1be9c211d2c1/vdis/5a7a5647-d2ca-4014-b87b-06c621c09a20/c6853f48-4b06-4c34-9707-b68f9054e6fc/20241007T111850Z.vhd'
Oct 7 11:47:30 xo-ce xo-server[11768]: }
Oct 7 11:47:30 xo-ce xo-server[11768]: 2024-10-07T11:47:30.568Z xo:backups:worker INFO backup has ended
Oct 7 11:48:00 xo-ce xo-server[11768]: 2024-10-07T11:48:00.630Z xo:backups:worker WARN worker process did not exit automatically, forcing...
Oct 7 11:48:00 xo-ce xo-server[11768]: 2024-10-07T11:48:00.631Z xo:backups:worker INFO process will exit {
Oct 7 11:48:00 xo-ce xo-server[11768]: duration: 856860228,
Oct 7 11:48:00 xo-ce xo-server[11768]: exitCode: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: resourceUsage: {
Oct 7 11:48:00 xo-ce xo-server[11768]: userCPUTime: 771667771,
Oct 7 11:48:00 xo-ce xo-server[11768]: systemCPUTime: 176351529,
Oct 7 11:48:00 xo-ce xo-server[11768]: maxRSS: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: sharedMemorySize: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: unsharedDataSize: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: unsharedStackSize: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: minorPageFault: 3322393,
Oct 7 11:48:00 xo-ce xo-server[11768]: majorPageFault: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: swappedOut: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: fsRead: 2416,
Oct 7 11:48:00 xo-ce xo-server[11768]: fsWrite: 209789336,
Oct 7 11:48:00 xo-ce xo-server[11768]: ipcSent: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: ipcReceived: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: signalsCount: 0,
Oct 7 11:48:00 xo-ce xo-server[11768]: voluntaryContextSwitches: 527248,
Oct 7 11:48:00 xo-ce xo-server[11768]: involuntaryContextSwitches: 342752
Oct 7 11:48:00 xo-ce xo-server[11768]: },
Oct 7 11:48:00 xo-ce xo-server[11768]: summary: { duration: '14m', cpuUsage: '111%', memoryUsage: '0 B' }
Oct 7 11:48:00 xo-ce xo-server[11768]: }
'uuidgen' is a linux tool that just automatically generates a unique identifier for the new .vhd file.
OK, found the git where it was added 2 months ago (https://github.com/vatesfr/xen-orchestra/pull/7308), and i haven't updated in 3 months, time to update XO 
hmm... i do not see such option. I forgot to mention that this is a delta backup job, if that matters.
Hi,
i have had this thing happen a few times now, where during scheduled backups, a VM starts migrating due to load balancing during a scheduled backup and then the backup job for that specific VM fails.
Error: cannot backup a VM currently being migrated
Is there a way to prevent this from happening, or at the very least have the backup job finish automatically after migration has finished?
Thanks
@AtaxyaNetwork can confirm this works for me as well
I feel like this should be default go-to method mentioned in the documentation and the current documentation mentioned as an alternative.
ps. im not a coder, no idea how to work github to make the changes my self
Hi,
I converted couple of .qcow2 images to .vmdk via qemu-img and imported the disks via XO attached the .vmdk to a new VM and the VM just booted and worked perfectly out of the box like that?
Reading the documentation here, the migration process should be much more complex but i didn't have to do any of that, is there some new magic happening that makes this work out of the box like that or should i expect issues later on doing the imports this way??
Also, after importing the .vmdk image to XO, i checked my SR and noticed that the image was automagically converted to .vhd and i cannot find any trace of the original imported .vmdk files, is this normal also?
Here are exact steps i took:
1. Sparify .qcow2 before conversion:
virt-sparsify image1.qcow2 sparse_image1.qcow2 --tmp /mnt/vmdata/backups/Server_name/2024-02-05.212740/
2. Do conversion:
qemu-img convert -p -f qcow2 -O vmdk sparse_image1.qcow2 image1.vmdk
3. Copied the .vmdk image to local machine and started disk import
4. Create VM with new disk
5. Remove created disk and attach imported .vmdk image
6. Boot up VM and everything works, Linux boots up done.
7. Checked SR on xcp-node and it is not .vhd file:
[13:08 xcp-node01 6b24cd1c-22ad-0994-5b6b-a75389a6ddba]# file 7e5ac555-68fd-406f-8b23-7185b3e30dab.vhd
7e5ac555-68fd-406f-8b23-7185b3e30dab.vhd: Microsoft Disk Image, Virtual Server or Virtual PC