Categories

  • All news regarding Xen and XCP-ng ecosystem

    143 Topics
    4k Posts
    I
    I am testing QCOW2. I do create 190 VMs on a single SR. The latency is growing with the count of VDIs on SR. [image: 1779297675818-3cbbd0d1-c3c1-4e9d-b74b-1933b6a64c51-image.jpeg] I got FC attached HPE AllFlash Storage. The other SR on same Storage has no performance impact. [image: 1779298232433-30494607-9e20-401f-b4c1-f04e03f0385d-image.jpeg] First 52 VMs was created without any problem. Than problems started. This VMs failed: #53 #94 #99 #100 #104 #106 #109 #110 #113 #115 #117 #120 #122 #124 #125 #128 #131 #136 #138 #145 #146 #152 #154 [... still pending] It loos like VDI.create for CloudConfigDrive runs into timeout. { "id": "0mpe928xl-jc1cgkgsmi", "properties": { "method": "vm.create", "params": { "name_label": "xcp-ng-vm156.lab.company.net", "name_description": "Test VM", "template": "0521468a-882b-9dd7-5add-2be588a389f8", "clone": false, "existingDisks": { "0": { "name_label": "xcp-ng-vm156.lab.company.net C", "$SR": "43dbbe8e-039a-e66c-f6cb-d88de4f4d962" } }, "CPUs": 4, "memory": 4294967296, "VIFs": [ { "network": "261a821c-a373-ba36-2c44-487ed0e1a202" } ], "cloudConfig": "**secret**", "coreOs": false, "bootAfterCreate": true, "destroyCloudConfigVdiAfterBoot": true, "tags": [ "win-cloud-init", "DUMMY" ], "hvmBootFirmware": "uefi" }, "name": "API call: vm.create", "userId": "033c2a08-7b45-4782-92f6-43acb73fe172", "type": "api.call" }, "start": 1779292916409, "status": "failure", "updatedAt": 1779294849770, "end": 1779294849770, "result": { "name": "HeadersTimeoutError", "code": "UND_ERR_HEADERS_TIMEOUT", "message": "Headers Timeout Error", "call": { "duration": 302136, "method": "VDI.create", "params": [ "* session id *", { "name_label": "XO CloudConfigDrive", "other_config": {}, "read_only": false, "sharable": false, "SR": "OpaqueRef:97a26f7e-0536-84de-f72c-49f11635df2d", "type": "user", "virtual_size": 10486272 } ] }, "stack": "HeadersTimeoutError: Headers Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/dispatcher/client-h1.js:642:28)\n at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:585:17)\n at processTimers (node:internal/timers:521:7)" } } The support tunnel is open, id: 40797 Feel free, it's lab environment.

  • Everything related to the virtualization platform

    1k Topics
    15k Posts
    D
    @AtaxyaNetwork no error but also no xe-linux-distribution service installed or available...

  • All Xen Orchestra related questions/reports

    3k Topics
    28k Posts
    acebmxerA
    @MathieuRA [ "/rest/v0/schedules/8cf0dcb7-2122-41c6-8a2c-5c150308b292", "/rest/v0/schedules/22ce7eb8-ea62-4cbd-8e9b-646f99b6896c", "/rest/v0/schedules/f3504435-9e2b-4e43-a4f4-2f70aa51aa30", "/rest/v0/schedules/d5f4478e-4bba-4679-be5b-375f7c9aeb09", "/rest/v0/schedules/77e3804e-2180-458a-969b-d08f44504c3a", "/rest/v0/schedules/aa8ee2fb-6587-4a5f-a7c7-f292f563b43c", "/rest/v0/schedules/70c6cfc3-baa6-4c8f-804b-ba273b14267d", "/rest/v0/schedules/da426cb4-9bc2-46bf-a5c6-268bb67b80ad", "/rest/v0/schedules/b4fa328a-856a-4e00-8e14-7353d45a79e1", "/rest/v0/schedules/5906f9f7-ca8f-4623-954c-744f0615fb55", "/rest/v0/schedules/3059d28e-eaf1-449f-a0d1-fece3712df0c", "/rest/v0/schedules/14b6af3b-5d2f-4bd2-9d80-440c1d773e2a", "/rest/v0/schedules/f45a8928-bf4b-4d36-bfb2-3d96735ef8fd" ] ❯ journalctl -u xo-server -f -n 50 May 21 07:20:45 xo-ce xo-server[8974]: } May 21 07:20:45 xo-ce xo-server[8974]: } May 21 07:20:45 xo-ce xo-server[8974]: 2026-05-21T07:20:45.705Z xo:xapi:xapi-disks INFO export through vhd May 21 07:20:47 xo-ce xo-server[8974]: 2026-05-21T07:20:47.589Z xo:xapi:vdi WARN invalid HTTP header in response body { May 21 07:20:47 xo-ce xo-server[8974]: body: 'HTTP/1.1 500 Internal Error\r\n' + May 21 07:20:47 xo-ce xo-server[8974]: 'content-length: 318\r\n' + May 21 07:20:47 xo-ce xo-server[8974]: 'content-type: text/html\r\n' + May 21 07:20:47 xo-ce xo-server[8974]: 'connection: close\r\n' + May 21 07:20:47 xo-ce xo-server[8974]: 'cache-control: no-cache, no-store\r\n' + May 21 07:20:47 xo-ce xo-server[8974]: '\r\n' + May 21 07:20:47 xo-ce xo-server[8974]: '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:fdeb8877-741a-d753-988f-34f20417f451; CBT metadata ]</body></html>' May 21 07:20:47 xo-ce xo-server[8974]: } May 21 07:20:47 xo-ce xo-server[8974]: 2026-05-21T07:20:47.598Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:e17293bd-788e-1391-4567-4a3b9f2be433 from OpaqueRef:fdeb8877-741a-d753-988f-34f20417f451, fallBack to a full { May 21 07:20:47 xo-ce xo-server[8974]: error: Error: invalid HTTP header in response body May 21 07:20:47 xo-ce xo-server[8974]: at checkVdiExport (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:37:19) May 21 07:20:47 xo-ce xo-server[8974]: at process.processTicksAndRejections (node:internal/process/task_queues:103:5) May 21 07:20:47 xo-ce xo-server[8974]: at async Xapi.exportContent (file:///opt/xen-orchestra/@xen-orchestra/xapi/vdi.mjs:261:5) May 21 07:20:47 xo-ce xo-server[8974]: at async #getExportStream (file:///opt/xen-orchestra/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20) May 21 07:20:47 xo-ce xo-server[8974]: at async XapiVhdStreamSource.init (file:///opt/xen-orchestra/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23) May 21 07:20:47 xo-ce xo-server[8974]: at async #openExportStream (file:///opt/xen-orchestra/@xen-orchestra/xapi/disks/Xapi.mjs:182:7) May 21 07:20:47 xo-ce xo-server[8974]: at async #openNbdStream (file:///opt/xen-orchestra/@xen-orchestra/xapi/disks/Xapi.mjs:97:22) May 21 07:20:47 xo-ce xo-server[8974]: at async XapiDiskSource.openSource (file:///opt/xen-orchestra/@xen-orchestra/xapi/disks/Xapi.mjs:260:18) May 21 07:20:47 xo-ce xo-server[8974]: at async XapiDiskSource.init (file:///opt/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41) May 21 07:20:47 xo-ce xo-server[8974]: at async file:///opt/xen-orchestra/@xen-orchestra/backups/_incrementalVm.mjs:66:5 May 21 07:20:47 xo-ce xo-server[8974]: } May 21 07:20:47 xo-ce xo-server[8974]: 2026-05-21T07:20:47.600Z xo:xapi:xapi-disks INFO export through vhd May 21 07:33:57 xo-ce xo-server[8974]: 2026-05-21T07:33:57.365Z xo:backups:worker INFO backup has ended May 21 07:33:57 xo-ce xo-server[8974]: 2026-05-21T07:33:57.558Z xo:backups:worker INFO process will exit { May 21 07:33:57 xo-ce xo-server[8974]: duration: 2036887755, May 21 07:33:57 xo-ce xo-server[8974]: exitCode: 0, May 21 07:33:57 xo-ce xo-server[8974]: resourceUsage: { May 21 07:33:57 xo-ce xo-server[8974]: userCPUTime: 2331929487, May 21 07:33:57 xo-ce xo-server[8974]: systemCPUTime: 555709233, May 21 07:33:57 xo-ce xo-server[8974]: maxRSS: 273572, May 21 07:33:57 xo-ce xo-server[8974]: sharedMemorySize: 0, May 21 07:33:57 xo-ce xo-server[8974]: unsharedDataSize: 0, May 21 07:33:57 xo-ce xo-server[8974]: unsharedStackSize: 0, May 21 07:33:57 xo-ce xo-server[8974]: minorPageFault: 11509457, May 21 07:33:57 xo-ce xo-server[8974]: majorPageFault: 3, May 21 07:33:57 xo-ce xo-server[8974]: swappedOut: 0, May 21 07:33:57 xo-ce xo-server[8974]: fsRead: 21664, May 21 07:33:57 xo-ce xo-server[8974]: fsWrite: 0, May 21 07:33:57 xo-ce xo-server[8974]: ipcSent: 0, May 21 07:33:57 xo-ce xo-server[8974]: ipcReceived: 0, May 21 07:33:57 xo-ce xo-server[8974]: signalsCount: 0, May 21 07:33:57 xo-ce xo-server[8974]: voluntaryContextSwitches: 3572469, May 21 07:33:57 xo-ce xo-server[8974]: involuntaryContextSwitches: 248999 May 21 07:33:57 xo-ce xo-server[8974]: }, May 21 07:33:57 xo-ce xo-server[8974]: summary: { duration: '34m', cpuUsage: '142%', memoryUsage: '267.16 MiB' } May 21 07:33:57 xo-ce xo-server[8974]: } ❯ journalctl -u xo-server May 03 18:14:41 xo-ce systemd[1]: Starting xo-server.service - Xen Orchestra Server... May 03 18:14:41 xo-ce systemd[1]: Started xo-server.service - Xen Orchestra Server. May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.248Z xo:main INFO Starting xo-server v5.200.0 (https://github.com> May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.255Z xo:main INFO Configuration loaded. May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.260Z xo:main INFO Web server listening on http://0.0.0.0:80 May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.263Z xo:main INFO Web server listening on https://0.0.0.0:443 May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.473Z xo:mixins:hooks WARN start failure { May 03 18:14:43 xo-ce xo-server[6780]: error: Error: spawn xenstore-read ENOENT May 03 18:14:43 xo-ce xo-server[6780]: at Process.ChildProcess._handle.onexit (node:internal/child_process:287:19) May 03 18:14:43 xo-ce xo-server[6780]: at onErrorNT (node:internal/child_process:508:16) May 03 18:14:43 xo-ce xo-server[6780]: at processTicksAndRejections (node:internal/process/task_queues:90:21) { May 03 18:14:43 xo-ce xo-server[6780]: errno: -2, May 03 18:14:43 xo-ce xo-server[6780]: code: 'ENOENT', May 03 18:14:43 xo-ce xo-server[6780]: syscall: 'spawn xenstore-read', May 03 18:14:43 xo-ce xo-server[6780]: path: 'xenstore-read', May 03 18:14:43 xo-ce xo-server[6780]: spawnargs: [ 'vm' ], May 03 18:14:43 xo-ce xo-server[6780]: cmd: 'xenstore-read vm' May 03 18:14:43 xo-ce xo-server[6780]: } May 03 18:14:43 xo-ce xo-server[6780]: } May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.478Z xo:mixins:hooks WARN start failure { May 03 18:14:43 xo-ce xo-server[6780]: error: Error: Command failed with exit code 5: vgchange -an May 03 18:14:43 xo-ce xo-server[6780]: WARNING: Running as a non-root user. Functionality may be unavailable. May 03 18:14:43 xo-ce xo-server[6780]: /run/lock/lvm/P_global:aux: open failed: Permission denied May 03 18:14:43 xo-ce xo-server[6780]: at makeError (file:///opt/xen-orchestra/packages/xo-server/node_modules/execa/l> May 03 18:14:43 xo-ce xo-server[6780]: at handlePromise (file:///opt/xen-orchestra/packages/xo-server/node_modules/exe> May 03 18:14:43 xo-ce xo-server[6780]: shortMessage: 'Command failed with exit code 5: vgchange -an', May 03 18:14:43 xo-ce xo-server[6780]: command: 'vgchange -an', May 03 18:14:43 xo-ce xo-server[6780]: escapedCommand: 'vgchange -an', May 03 18:14:43 xo-ce xo-server[6780]: exitCode: 5, May 03 18:14:43 xo-ce xo-server[6780]: signal: undefined, May 03 18:14:43 xo-ce xo-server[6780]: signalDescription: undefined, May 03 18:14:43 xo-ce xo-server[6780]: stdout: '', May 03 18:14:43 xo-ce xo-server[6780]: stderr: ' WARNING: Running as a non-root user. Functionality may be unavailable.> May 03 18:14:43 xo-ce xo-server[6780]: ' /run/lock/lvm/P_global:aux: open failed: Permission denied', May 03 18:14:43 xo-ce xo-server[6780]: cwd: '/opt/xen-orchestra/packages/xo-server', May 03 18:14:43 xo-ce xo-server[6780]: failed: true, May 03 18:14:43 xo-ce xo-server[6780]: timedOut: false, May 03 18:14:43 xo-ce xo-server[6780]: isCanceled: false, May 03 18:14:43 xo-ce xo-server[6780]: killed: false May 03 18:14:43 xo-ce xo-server[6780]: } May 03 18:14:43 xo-ce xo-server[6780]: } May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.556Z xo:xo-mixins:subjects INFO Default user created: admin@admin> May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.565Z xo:main INFO Setting up /robots.txt → /opt/xen-orchestra/pac> May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.616Z xo:plugin INFO register audit May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.618Z xo:plugin INFO register auth-github May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.619Z xo:plugin INFO register auth-google May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.620Z xo:plugin INFO register auth-ldap May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.621Z xo:plugin INFO register auth-oidc May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.621Z xo:plugin INFO register auth-saml May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.622Z xo:plugin INFO register backup-reports May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.623Z xo:plugin INFO register load-balancer May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.623Z xo:plugin INFO register netbox May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.624Z xo:plugin INFO register openmetrics May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.625Z xo:plugin INFO register perf-alert May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.626Z xo:plugin INFO register sdn-controller May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.627Z xo:plugin INFO register test-plugin May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.628Z xo:plugin INFO register transport-email May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.628Z xo:plugin INFO register transport-icinga2 May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.629Z xo:plugin INFO register transport-nagios May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.629Z xo:plugin INFO register transport-slack May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.630Z xo:plugin INFO register transport-xmpp May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.630Z xo:plugin INFO register usage-report May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.631Z xo:plugin INFO register web-hooks May 03 18:14:43 xo-ce xo-server[6780]: 2026-05-03T18:14:43.998Z xo:xo-server-openmetrics INFO Plugin initialized May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin auth-gith> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin audit for> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin auth-oidc> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin auth-goog> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin auth-ldap> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin auth-saml> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin load-bala> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin netbox fo> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin perf-aler> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin sdn-contr> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin backup-re> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin test-plug> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin transport> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin transport> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin transport> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin transport> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin transport> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin web-hooks> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin usage-rep> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.012Z xo:xo-mixins:plugins INFO [NOTICE] register plugin openmetri> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.027Z xo:plugin INFO successfully register auth-github May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.027Z xo:plugin INFO successfully register auth-oidc May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.027Z xo:plugin INFO successfully register auth-google May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.027Z xo:plugin INFO successfully register auth-ldap May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.027Z xo:plugin INFO successfully register auth-saml May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.027Z xo:plugin INFO successfully register load-balancer May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.040Z xo:plugin INFO successfully register audit May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register netbox May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register test-plugin May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register transport-nagios May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register transport-email May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register transport-icinga2 May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register transport-slack May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register transport-xmpp May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register web-hooks May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:plugin INFO successfully register usage-report May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.056Z xo:xo-server-openmetrics INFO Starting OpenMetrics server { > May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.062Z xo:plugin INFO successfully register backup-reports May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.062Z xo:plugin INFO successfully register perf-alert May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.142Z xo:plugin INFO successfully register openmetrics May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.480Z xo:plugin INFO successfully register sdn-controller May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.481Z xo:main INFO Setting up /v5 → /opt/xen-orchestra/packages/xo> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.481Z xo:main INFO Setting up /v6 → /opt/xen-orchestra/@xen-orches> May 03 18:14:44 xo-ce xo-server[6780]: 2026-05-03T18:14:44.481Z xo:main INFO Setting up / → /opt/xen-orchestra/@xen-orchestr> May 03 18:15:16 xo-ce xo-server[6780]: (node:6780) [DEP0060] DeprecationWarning: The `util._extend` API is deprecated. Pleas> May 03 18:15:16 xo-ce xo-server[6780]: (Use `node --trace-deprecation ...` to show where the warning was created) May 03 18:15:16 xo-ce xo-server[6780]: 2026-05-03T18:15:16.063Z xo:xo-server WARN Node warning { May 03 18:15:16 xo-ce xo-server[6780]: error: DeprecationWarning: The `util._extend` API is deprecated. Please use Object.> May 03 18:15:16 xo-ce xo-server[6780]: at ProxyServer.ws (/opt/xen-orchestra/node_modules/http-proxy/lib/http-proxy/in> May 03 18:15:16 xo-ce xo-server[6780]: at app.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/index.mjs:> May 03 18:15:16 xo-ce xo-server[6780]: at app.emit (node:events:509:28) May 03 18:15:16 xo-ce xo-server[6780]: at app.emit (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17) May 03 18:15:16 xo-ce xo-server[6780]: at Server.<anonymous> (file:///opt/xen-orchestra/packages/xo-server/src/index.m> May 03 18:15:16 xo-ce xo-server[6780]: at Server.emit (node:events:521:24) May 03 18:15:16 xo-ce xo-server[6780]: at Server.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configur> May 03 18:15:16 xo-ce xo-server[6780]: at onParserExecuteCommon (node:_http_server:987:14) May 03 18:15:16 xo-ce xo-server[6780]: at onParserExecute (node:_http_server:891:3) { May 03 18:15:16 xo-ce xo-server[6780]: code: 'DEP0060' May 03 18:15:16 xo-ce xo-server[6780]: } May 03 18:15:16 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: 2026-05-03T18:15:30.553Z xo:rest-api:listener ERROR cannot handle data for task liste> May 03 18:15:30 xo-ce xo-server[6780]: error: XoError: no such user a7eed653-d733-4909-99b6-88afb4baf6c4 May 03 18:15:30 xo-ce xo-server[6780]: at noSuchObject (/opt/xen-orchestra/packages/xo-common/api-errors.js:26:11) May 03 18:15:30 xo-ce xo-server[6780]: at default._getUser (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins> May 03 18:15:30 xo-ce xo-server[6780]: at default.getUser (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/> May 03 18:15:30 xo-ce xo-server[6780]: at XoListener.getAclEvent (file:///opt/xen-orchestra/@xen-orchestra/rest-api/di> May 03 18:15:30 xo-ce xo-server[6780]: at XoListener.handleData (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dis> May 03 18:15:30 xo-ce xo-server[6780]: at file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/abstract-classes/list> May 03 18:15:30 xo-ce xo-server[6780]: code: 1, May 03 18:15:30 xo-ce xo-server[6780]: data: { id: 'a7eed653-d733-4909-99b6-88afb4baf6c4', type: 'user' } May 03 18:15:30 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: 2026-05-03T18:15:30.553Z xo:rest-api:listener ERROR cannot handle data for task liste> May 03 18:15:30 xo-ce xo-server[6780]: error: XoError: no such user a7eed653-d733-4909-99b6-88afb4baf6c4 May 03 18:15:30 xo-ce xo-server[6780]: at noSuchObject (/opt/xen-orchestra/packages/xo-common/api-errors.js:26:11) May 03 18:15:30 xo-ce xo-server[6780]: at default._getUser (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins> May 03 18:15:30 xo-ce xo-server[6780]: at default.getUser (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/> May 03 18:15:30 xo-ce xo-server[6780]: at XoListener.getAclEvent (file:///opt/xen-orchestra/@xen-orchestra/rest-api/di> May 03 18:15:30 xo-ce xo-server[6780]: at XoListener.handleData (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dis> May 03 18:15:30 xo-ce xo-server[6780]: at file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/abstract-classes/list> May 03 18:15:30 xo-ce xo-server[6780]: code: 1, May 03 18:15:30 xo-ce xo-server[6780]: data: { id: 'a7eed653-d733-4909-99b6-88afb4baf6c4', type: 'user' } May 03 18:15:30 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: 2026-05-03T18:15:30.557Z xo:authentification ERROR no such user a7eed653-d733-4909-99> May 03 18:15:30 xo-ce xo-server[6780]: error: XoError: no such user a7eed653-d733-4909-99b6-88afb4baf6c4 May 03 18:15:30 xo-ce xo-server[6780]: at noSuchObject (/opt/xen-orchestra/packages/xo-common/api-errors.js:26:11) May 03 18:15:30 xo-ce xo-server[6780]: at default._getUser (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins> May 03 18:15:30 xo-ce xo-server[6780]: at default.getUser (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/> May 03 18:15:30 xo-ce xo-server[6780]: at default._authenticateUser (file:///opt/xen-orchestra/packages/xo-server/src/> May 03 18:15:30 xo-ce xo-server[6780]: at auth (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/authenticat> May 03 18:15:30 xo-ce xo-server[6780]: at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22) May 03 18:15:30 xo-ce xo-server[6780]: at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20) May 03 18:15:30 xo-ce xo-server[6780]: at Xo.signIn (file:///opt/xen-orchestra/packages/xo-server/src/api/session.mjs:> May 03 18:15:30 xo-ce xo-server[6780]: at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:204:22) May 03 18:15:30 xo-ce xo-server[6780]: at Task.run (/opt/xen-orchestra/@vates/task/index.js:188:20) May 03 18:15:30 xo-ce xo-server[6780]: at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixi> May 03 18:15:30 xo-ce xo-server[6780]: code: 1, May 03 18:15:30 xo-ce xo-server[6780]: data: { id: 'a7eed653-d733-4909-99b6-88afb4baf6c4', type: 'user' } May 03 18:15:30 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: } May 03 18:15:30 xo-ce xo-server[6780]: 2026-05-03T18:15:30.560Z xo:api WARN (unknown user) | session.signIn(...) [8ms] =!> X> May 03 18:15:53 xo-ce sudo[6982]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) May 03 18:15:53 xo-ce sudo[6982]: xo-service : PWD=/opt/xen-orchestra/packages/xo-server ; USER=root ; COMMAND=/usr/bin/mou> May 03 18:15:53 xo-ce sudo[6984]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) May 03 18:15:53 xo-ce sudo[6984]: xo-service : PWD=/opt/xen-orchestra/packages/xo-server ; USER=root ; COMMAND=/usr/bin/mou> May 03 18:15:54 xo-ce sudo[6984]: pam_unix(sudo:session): session closed for user root May 03 18:15:54 xo-ce sudo[6982]: pam_unix(sudo:session): session closed for user root May 03 18:16:45 xo-ce xo-server[6780]: 2026-05-03T18:16:45.301Z xo:main INFO + Console proxy (nick - 127.0.0.1) May 03 18:16:47 xo-ce xo-server[6780]: 2026-05-03T18:16:47.299Z xo:main INFO - Console proxy (nick - 127.0.0.1) May 03 23:04:20 xo-ce xo-server[6780]: 2026-05-03T23:04:20.943Z xo:api WARN nick | token.create(...) [1ms] =!> Error: too hi> May 03 23:04:38 xo-ce xo-server[6780]: 2026-05-03T23:04:38.359Z xo:api WARN nick | token.create(...) [1ms] =!> Error: too hi> May 03 23:04:52 xo-ce xo-server[6780]: 2026-05-03T23:04:52.662Z xo:api WARN nick | token.create(...) [1ms] =!> Error: too hi> May 03 23:05:19 xo-ce xo-server[6780]: 2026-05-03T23:05:19.432Z xo:api WARN nick | token.create(...) [1ms] =!> Error: too hi> May 03 23:34:52 xo-ce xo-server[6780]: 2026-05-03T23:34:52.293Z xo:main INFO + Console proxy (nick - 127.0.0.1) May 03 23:36:04 xo-ce xo-server[6780]: 2026-05-03T23:36:04.989Z xo:main INFO - Console proxy (nick - 127.0.0.1) May 04 04:08:33 xo-ce xo-server[8919]: 2026-05-04T04:08:33.617Z xo:backups:worker INFO starting backup May 04 04:08:33 xo-ce sudo[8932]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) May 04 04:08:33 xo-ce sudo[8932]: xo-service : PWD=/opt/xen-orchestra/packages/xo-server ; USER=root ; COMMAND=/usr/bin/mou> May 04 04:08:33 xo-ce sudo[8932]: pam_unix(sudo:session): session closed for user root May 04 04:08:36 xo-ce xo-server[8919]: 2026-05-04T04:08:36.432Z xo:xapi:xapi-disks INFO export through vhd May 04 04:08:37 xo-ce xo-server[8919]: 2026-05-04T04:08:37.874Z xo:xapi:xapi-disks INFO export through vhd May 04 04:12:16 xo-ce xo-server[8919]: 2026-05-04T04:12:16.797Z xo:xapi:xapi-disks INFO export through vhd May 04 04:37:10 xo-ce xo-server[8919]: 2026-05-04T04:37:10.906Z xo:xapi:xapi-disks INFO export through vhd May 04 04:46:44 xo-ce xo-server[8919]: { May 04 04:46:44 xo-ce xo-server[8919]: err: [Error: Unknown system error -122: Unknown system error -122, write] { May 04 04:46:44 xo-ce xo-server[8919]: errno: -122, May 04 04:46:44 xo-ce xo-server[8919]: code: 'Unknown system error -122', May 04 04:46:44 xo-ce xo-server[8919]: syscall: 'write' May 04 04:46:44 xo-ce xo-server[8919]: } May 04 04:46:44 xo-ce xo-server[8919]: } May 04 04:46:44 xo-ce xo-server[8919]: 2026-05-04T04:46:44.907Z xo:backups:AbstractVmRunner WARN writer step failed { May 04 04:46:44 xo-ce xo-server[8919]: error: Error: ENOENT: no such file or directory, open '/run/xo-server/mounts/3ba9ee> May 04 04:46:44 xo-ce xo-server[8919]: From: May 04 04:46:44 xo-ce xo-server[8919]: at NfsHandler.addSyncStackTrace (/opt/xen-orchestra/@xen-orchestra/fs/dist/loca> May 04 04:46:44 xo-ce xo-server[8919]: at NfsHandler._openFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:154> May 04 04:46:44 xo-ce xo-server[8919]: at /opt/xen-orchestra/@xen-orchestra/fs/dist/utils.js:29:26 May 04 04:46:44 xo-ce xo-server[8919]: at new Promise (<anonymous>) May 04 04:46:44 xo-ce xo-server[8919]: at NfsHandler.<anonymous> (/opt/xen-orchestra/@xen-orchestra/fs/dist/utils.js:2> May 04 04:46:44 xo-ce xo-server[8919]: at loopResolver (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:83:46) May 04 04:46:44 xo-ce xo-server[8919]: at new Promise (<anonymous>) May 04 04:46:44 xo-ce xo-server[8919]: at loop (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:85:22) May 04 04:46:44 xo-ce xo-server[8919]: at NfsHandler.retry (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:8> May 04 04:46:44 xo-ce xo-server[8919]: at NfsHandler._openFile (/opt/xen-orchestra/node_modules/promise-toolbox/retry.> May 04 04:46:44 xo-ce xo-server[8919]: errno: -2, May 04 04:46:44 xo-ce xo-server[8919]: code: 'ENOENT', May 04 04:46:44 xo-ce xo-server[8919]: syscall: 'open', May 04 04:46:44 xo-ce xo-server[8919]: path: '/run/xo-server/mounts/3ba9eed2-55f1-4760-ac08-655c252aa316/xo-vm-backups/f> May 04 04:46:44 xo-ce xo-server[8919]: }, May 04 04:46:44 xo-ce xo-server[8919]: step: 'writer.updateUuidAndChain()', May 04 04:46:44 xo-ce xo-server[8919]: writer: 'IncrementalRemoteWriter' May 04 04:46:44 xo-ce xo-server[8919]: } May 04 04:46:51 xo-ce xo-server[8919]: 2026-05-04T04:46:51.605Z xo:xapi:xapi-disks INFO export through vhd May 04 04:46:51 xo-ce xo-server[8919]: 2026-05-04T04:46:51.606Z xo:xapi:xapi-disks INFO export through vhd May 04 04:46:53 xo-ce xo-server[8919]: { May 04 04:46:53 xo-ce xo-server[8919]: err: [Error: Unknown system error -122: Unknown system error -122, write] { May 04 04:46:53 xo-ce xo-server[8919]: errno: -122, May 04 04:46:53 xo-ce xo-server[8919]: code: 'Unknown system error -122', May 04 04:46:53 xo-ce xo-server[8919]: syscall: 'write' May 04 04:46:53 xo-ce xo-server[8919]: } May 04 04:46:53 xo-ce xo-server[8919]: } May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.434Z vates:nbd-client WARN will retry reading block 13 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 14 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 15 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 16 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 17 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 18 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 19 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 20 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z vates:nbd-client WARN will retry reading block 21 May 04 04:46:53 xo-ce xo-server[8919]: 2026-05-04T04:46:53.435Z xo:backups:AbstractVmRunner WARN writer step failed { May 04 04:46:53 xo-ce xo-server[8919]: error: Error: ENOENT: no such file or directory, open '/run/xo-server/mounts/3ba9ee> May 04 04:46:53 xo-ce xo-server[8919]: From: May 04 04:46:53 xo-ce xo-server[8919]: at NfsHandler.addSyncStackTrace (/opt/xen-orchestra/@xen-orchestra/fs/dist/loca> May 04 04:46:53 xo-ce xo-server[8919]: at NfsHandler._openFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:154> May 04 04:46:53 xo-ce xo-server[8919]: at /opt/xen-orchestra/@xen-orchestra/fs/dist/utils.js:29:26 May 04 04:46:53 xo-ce xo-server[8919]: at new Promise (<anonymous>) May 04 04:46:53 xo-ce xo-server[8919]: at NfsHandler.<anonymous> (/opt/xen-orchestra/@xen-orchestra/fs/dist/utils.js:2> May 04 04:46:53 xo-ce xo-server[8919]: at loopResolver (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:83:46) May 04 04:46:53 xo-ce xo-server[8919]: at new Promise (<anonymous>) May 04 04:46:53 xo-ce xo-server[8919]: at loop (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:85:22) May 04 04:46:53 xo-ce xo-server[8919]: at NfsHandler.retry (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:8> May 04 04:46:53 xo-ce xo-server[8919]: at NfsHandler._openFile (/opt/xen-orchestra/node_modules/promise-toolbox/retry.> May 04 04:46:53 xo-ce xo-server[8919]: errno: -2, May 04 04:46:53 xo-ce xo-server[8919]: code: 'ENOENT', May 04 04:46:53 xo-ce xo-server[8919]: syscall: 'open', May 04 04:46:53 xo-ce xo-server[8919]: path: '/run/xo-server/mounts/3ba9eed2-55f1-4760-ac08-655c252aa316/xo-vm-backups/4> May 04 04:46:53 xo-ce xo-server[8919]: }, May 04 04:46:53 xo-ce xo-server[8919]: step: 'writer.updateUuidAndChain()', May 04 04:46:53 xo-ce xo-server[8919]: writer: 'IncrementalRemoteWriter' May 04 04:46:53 xo-ce xo-server[8919]: } May 04 04:46:58 xo-ce xo-server[8919]: 2026-05-04T04:46:58.298Z xo:xapi WARN retry { May 04 04:46:58 xo-ce xo-server[8919]: attemptNumber: 0, May 04 04:46:58 xo-ce xo-server[8919]: delay: 5000, May 04 04:46:58 xo-ce xo-server[8919]: error: XapiError: VDI_IN_USE(OpaqueRef:50dc4331-ae9e-5cfe-fbb0-d5f60e311ee6, data_d> May 04 04:46:58 xo-ce xo-server[8919]: at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16>

  • Our hyperconverged storage solution

    47 Topics
    745 Posts
    J
    @Mathieu-L linstor n l was included in my original post. All nodes were updated to May 2026 Security and Maintenance Updates for XCP-ng 8.3 LTS, all nodes were restarted. May 2026 Updates #2 for XCP-ng 8.3 LTS was released, and a couple days later I installed on all hosts. No host restarted. When xen04 was restarted, that is when this issue happened. I had used systemctl restart linstor-controller here (https://xcp-ng.org/forum/post/105309) to restart the controller.

  • Non-English speakers

    35 Topics
    113 Posts
    olivierlambertO
    Ah excellente nouvelle Je passe le sujet en résolu !