XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    • Profile
    • Following 0
    • Followers 0
    • Topics 11
    • Posts 80
    • Groups 0
    J Offline
    1. Home
    2. jr-m4
    3. Posts

    Posts

    Recent Best Controversial
    • RE: A VM without any data on disk, passes Health Check - Bug?

      @poddingue

      Thanks for taking a look on your end as well.
      I would like to point out that the line from the docs is quite deffinitive. If there are gotchas, such as UEFI affecting things. Then this either should be documented. Or better yet, that outlier of an edge-case probably should be addressed.

      One scenario I'm envisioning is that a disk is faulty for some reason, and this goes unoticed due to the VM being rarely used. And since no backup-errors are given, this might go unoticed for quite some time. The Health Check passing is a false signal in that case.

      These are my two-cents. And I confirm that this is an edge-case, and even unlikely one. But a theoretical greater than zero risk anyways.

      Thank you

      posted in Backup
      J
      jr-m4
    • A VM without any data on disk, passes Health Check - Bug?

      After doing some reshuffling and having to re-create a couple of VMs. I have two with newly created disks, that don't contain any data yet.
      Upon doing some checks on the backup-schedules, I notice that these two passed the latest Delta-backup Health Check. However, as I mentioned earlier. These don't have data on their disks, and hence no Guest-Tools can be installed.

      "A VM without guest tools will fail its health check."

      source: https://docs.xen-orchestra.com/backups-and-dr/backup-features-and-settings#backup-health-check

      image.jpeg

      posted in Backup
      J
      jr-m4
    • RE: Support for newer Emulex LPe37xxx amd LPe36xxx Fiber channel adapters on XCP-ng 8.3 - your testing is needed

      Third update:
      It looks more and more that I had goofed something up in my testing. So please disregard everything in the post below.
      The driver does work to make our LPe37000/LPe38000 series FC-cards work. Which is nice. Thanks!

      ~~Good morning

      So I might've found an issue with this driver release. That I'm trying to troubleshoot and collect data on.

      We have two LUNs presented to our dev-pool (xcp-dev), running on R650xs. When I have attached one LUN to xcp-dev I can't see the other LUN.
      This works on our older R350s

      Removing the attached LUN, makes a LUN visible again. But now we're getting some strange behaviour. And looking into if it is misconfigured from the Dell Powerstore, or if it is on the host(s) itself.~~

      Details:
      multipath is enabled

      Host on xcp-dev

      98:00.0 Fibre Channel [0c04]: Emulex Corporation LPe37000/LPe38000 Series 32Gb/64Gb Fibre Channel Adapter [10df:f500] (rev 11)
              Subsystem: Emulex Corporation Device [10df:f522]
              Kernel driver in use: lpfc
              Kernel modules: lpfc14
      98:00.1 Fibre Channel [0c04]: Emulex Corporation LPe37000/LPe38000 Series 32Gb/64Gb Fibre Channel Adapter [10df:f500] (rev 11)
              Subsystem: Emulex Corporation Device [10df:f522]
              Kernel driver in use: lpfc
              Kernel modules: lpfc14
      

      Update:
      ~~After managing to revert my previous tests and forget the existing (wrong) LUN from xcp-dev. I can now confirm a problem.

      There are three LUNs. Let's call them 101, 102, 103

      When I attach one LUN (103) to xcp-dev, I am no longer able to see the other LUNs. That means I can't add a second one. I am expecting to be able to see 102 available.

      However, on old-dev, that has two LUNs attached already (101 & 102). It can see 103 as expected.~~

      host on old-dev

      02:00.0 Fibre Channel [0c04]: Emulex Corporation LPe12000 Series 8Gb Fibre Channel Adapter [10df:f100] (rev 03)
              Subsystem: Emulex Corporation LPe12000 Series 8Gb Fibre Channel Adapter [10df:f100]
              Kernel driver in use: lpfc
              Kernel modules: lpfc
      02:00.1 Fibre Channel [0c04]: Emulex Corporation LPe12000 Series 8Gb Fibre Channel Adapter [10df:f100] (rev 03)
              Subsystem: Emulex Corporation LPe12000 Series 8Gb Fibre Channel Adapter [10df:f100]
              Kernel driver in use: lpfc
              Kernel modules: lpfc
      

      Second Update:
      This might be unrelated. Seems that we're having weird behaviour with the LUNs in general. Please disregard the statement above for now

      posted in Hardware
      J
      jr-m4
    • RE: [V2V] Without VDDK problems

      @mpiton

      Thanks for a fantasticly quick answer, and detailed as well. I will test this immediately!

      Update:
      I can confirm that this was indeed the problem!

      posted in Migrate to XCP-ng
      J
      jr-m4
    • RE: [V2V] Without VDDK problems

      Looking into journcelctl on the XO-server, after I enabled CBT-tracking on the VM (using this KB from Broadcom https://knowledge.broadcom.com/external/article/315370/enabling-or-disabling-changed-block-trac.html). I get the following:

      The initial V2V-snapshot on vmware is created as expected. So there is communication between XO and vmware (vcenter-host)

      Oct 02 13:24:33 hostname-xo-server xo-server[55639]: 2026-10-02T11:24:33.361Z xo:vmware-explorer:esxi INFO got the data map from the change tracking of the host {
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]:   blocks: 16,
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]:   bytes: 20912799744,
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]:   diskPath: 'vm-name_1/vm-name.vmdk',
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]:   full: true,
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]:   seconds: 0,
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]:   vmId: 'vm-1169935'
      Oct 02 13:24:33 hostname-xo-server xo-server[55639]: }
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]: 2026-10-02T11:24:44.496Z vates:nbd-client:stdio WARN the nbd server process exited unexpectedly {
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:   error: Error: the nbd server process exited (code: 1, signal: null)
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:       at ChildProcess.<anonymous> (file:///opt/xen-orchestra/@vates/nbd-client/NbdStdioClient.mjs:107:21)
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:       at ChildProcess.wrapper (node:events:639:12)
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:       at ChildProcess.emit (node:events:514:28)
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:       at ChildProcess.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:       at Process.ChildProcess._handle.onexit (node:internal/child_process:295:12) {
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:     code: 'NBD_SERVER_EXITED',
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:     exitCode: 1,
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:     signal: null,
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:     command: '/opt/xen-orchestra/@xen-orchestra/vmware-explorer/vectura/vectura',
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:     stderr: 'vectura: data plane: could not connect: timed out\n'
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]:   }
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]: }
      Oct 02 13:24:44 hostname-xo-server xo-server[55639]: 2026-10-02T11:24:44.526Z xo:api WARN username-1 | vm.importMultipleFromEsxi(...) [17s] =!> Error: stream has ended without data, was looking for 8 bytes
      
      posted in Migrate to XCP-ng
      J
      jr-m4
    • [V2V] Without VDDK problems

      First of all. Thanks for putting alot of effort into getting migrations to work without VDDK. https://github.com/vatesfr/xen-orchestra/pull/10421

      However. I did a new provisioning of an XO-source instance. And wanted to test it without any hanging dependencies from previously installed vddk's.

      Now I'm getting met with failed import-attempts.

      Attempts with my previous XO instance, on the same commit: b59c8 works just fine. Except the VMs not starting automatically after import (discussed in separate thread: https://xcp-ng.org/forum/post/108686)

      vm.importMultipleFromEsxi
      {
        "concurrency": 2,
        "host": "vcenter-host",
        "network": "72471fb1-58fd-d2d1-9a1e-46aea05374b4",
        "password": "* obfuscated *",
        "sr": "4d58d7f1-087a-bc7b-a957-e8f8be8b1f77",
        "sslVerify": false,
        "stopOnError": true,
        "stopSource": true,
        "template": "37e7a3b9-8c45-c7f2-7d09-249a935dd33d-1a647cf9-99c1-4e9c-b3b4-6b9989c530be",
        "user": "obfuscated",
        "vms": [
          "vm-1169935"
        ]
      }
      {
        "succeeded": {},
        "message": "stream has ended without data, was looking for 8 bytes",
        "name": "Error",
        "stack": "Error: stream has ended without data, was looking for 8 bytes
          at readChunkStrict (/opt/xen-orchestra/@vates/read-chunk/index.js:83:13)"
      }
      “““
      posted in Migrate to XCP-ng
      J
      jr-m4
    • RE: XCP-ng 8.3 updates announcements and testing

      Installed on 3xDell PowerEdge R350 in HA. And so far, I haven't noticed anything out of the ordinary.

      posted in News
      J
      jr-m4
    • RE: Enable Maintenance Mode = Host Not Enough Memory

      @poddingue
      Our HA-pool is a 3 host system, yes.

      I'll add a note about our plans to go paid in that feedback-item. Thanks for the tip.
      In the meantime, I'm testing and reporting as much as I can. In order to hopefully help the product be better for all.

      Cheers!

      posted in XCP-ng
      J
      jr-m4
    • RE: Enable Maintenance Mode = Host Not Enough Memory

      @poddingue

      Huh.. Good catch.
      I set all of the VMs to restart, and now it does make maintenance mode possible, on the host.

      However, when we go into licensed/paid production (soon tm). It would be an absolute nightmare to try and find one VM that might not be on the right HA-plan... And it does kind of feel that Maintenance-mode should be compatible with whatever HA-plan is chosen regardless.

      Perhaps there could be a way to assign a global/pool-wide HA-plan along with enabling HA alltogether?

      Update: Shining light on that there is already a feedback-item, wishing for this kind of mass-operation for HA.
      https://feedback.vates.tech/posts/45/bulk-operation-for-setting-vm-ha-option

      posted in XCP-ng
      J
      jr-m4
    • Enable Maintenance Mode = Host Not Enough Memory

      Doing some more testing, where I wished to place one Host in maintenance mode. I was then met with HOST_NOT_ENOUGH_FREE_MEMORY.
      The pool is a 3 host pool in HA, with load balancing enabled.

      My guess is that when trying to enable Maintenance mode. It tries to migrate ALL VMs on that host, to ONE single Host on the receiving end. And then it exhausts the amount of available memory.

      What I had to do, was manually disable load balancing. And manually migrate VMs to the two other hosts, to distribute them. This worked.

      What I expected to have happened:
      The evacuation of the host going into Maintenance mode, would be distributing the VMs in a smarter fashion. So that their workload would fit in the pooled resources available.

      Commit: 2f846
      XCP-NG: Fully updated as of writing

      host.setMaintenanceMode
      {
        "id": "<obfuscated>",
        "maintenance": true
      }
      {
        "code": "HOST_NOT_ENOUGH_FREE_MEMORY",
        "params": [
          "OpaqueRef:<obfuscated>"
        ],
        "task": {
          "uuid": "c3a83a2d-003d-ac0f-48b8-79a905c9e557",
          "name_label": "Async.host.evacuate",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20260929T07:46:40Z",
          "finished": "20260929T07:46:40Z",
          "status": "failure",
          "resident_on": "OpaqueRef:<obfuscated>",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "HOST_NOT_ENOUGH_FREE_MEMORY",
            "OpaqueRef:<obfuscated>"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 629))((process xapi)(filename hashtbl.ml)(line 159))((process xapi)(filename hashtbl.ml)(line 165))((process xapi)(filename hashtbl.ml)(line 170))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 625))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
        },
        "message": "HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:<obfuscated>)",
        "name": "XapiError",
        "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:<obfuscated>)
          at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
          at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
          at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1358:24)
          at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1392:14
          at Array.forEach (<anonymous>)
          at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1382:12)
          at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1589:14)"
      }
      

      Update: Edited title, since it would suggest Load Balancing was the problem. It isn't. It's the assignment of migration target(s) that is the issue (imho)

      posted in XCP-ng
      J
      jr-m4
    • RE: V2V migrated VMs don't autostart

      @poddingue

      Thanks for taking a look.

      Yes it is this part that tells me that it should start the VM automatically once the import has completed.
      "After the transfer, the VM on XCP-ng side is started:"
      Followed by, "This process is fully automated, without any human intervention after it starts on step 1."
      image.jpeg
      https://docs.xcp-ng.org/installation/migrate-to-xcp-ng/

      But my findings/experience is that the VM does not start once the import/transfer has completed.

      So either the docs are wrong. Or the function is not behaving as intended. I'm acctually fine either way. As long as I know what to expect. However, I would like to have it auto-start the VMs after transfer completion. Which would make the procedure "Fully automated".

      Update:
      Am I assuming correctly that "the XO V2V guide says to start the VM yourself once the migration is complete " you're reffering to, is part of the "test migration"-procedure? Not the acctual production mgiration?
      Because:

      "Final migration
      Run the production migration
      When you're ready for the final migration:
      
      Shut down the source VM completely.
      Start the V2V migration in Xen Orchestra with the Stop source option enabled.
      This ensures the final sync happens while the VM is powered off, and prevents any inconsistencies."
      

      In my mind doesn't state that the VM should be started manually post-transfer.

      Great thanks in advance for discussing!
      Cheers!

      posted in Migrate to XCP-ng
      J
      jr-m4
    • RE: V2V migrated VMs don't autostart

      Further testing:

      Testing with newer commit: 27882fb1dcd24fc5257835e60b3fa1002e4e885c
      Still same results.

      I've tried with both original VM powered on, and off. No change in end-result. The VM imports properly. But it doesn't power on automatically as expected after the import has completed. Manual powering on is then needed to be performed.

      I'm only importing 1 VM at a time. And I've tried with the field Number of VMs to import in parallel both at the default value of 2 and reduced to 1. No change in behaviour from this either.

      posted in Migrate to XCP-ng
      J
      jr-m4
    • RE: V2V migrated VMs don't autostart

      @tjkreidl

      Thanks for your long post. I hope someone else will find it helpfull.
      However, I am pretty much reporting that the autostart after importing with V2V might be buggy at best or broken at worst. Not the autostart after boot part šŸ™‚ (that works).

      Cheers

      posted in Migrate to XCP-ng
      J
      jr-m4
    • RE: Reducing vCPU isn't done live on Windows

      @dinhngtu said:

      Hello,

      Windows only supports adding CPUs but not removing them.

      This is a Windows limitation.

      I'm not surprised by that limitation. Thanks for confirming anyway.

      Cheers!

      posted in XCP-ng
      J
      jr-m4
    • V2V migrated VMs don't autostart

      I'm at the moment doing alot of test with V2V (applause on getting rid of the vddk-dependency btw šŸ˜„ )

      However, what I'm noticing is that the migrated VMs don't automatically start when the import has completed. This is contrary to current documentation.

      Is this a potential bug? Or am I missing something obvious?

      PS: The imports are all sucessfull. It's just the auto-start part that I'm noticing doesn't work.

      Cheers!

      Commit: dc573

      Template used for testig is: Rocky Linux 9

      posted in Migrate to XCP-ng
      J
      jr-m4
    • Reducing vCPU isn't done live on Windows

      I'm doing a couple of tests to trying to figure out quirks and behaviours from XCP-ng side, for a future production (licensed) setup.

      So I tried to see what would happen if I would play with the number of vCPUs on two guests.
      TL;DR: Reducing vCPUs on Windows requires reboot.

      On Windows Guest:
      OS: Windows Server 25
      Guest Agent: 9.2.385.0
      Increasing the number of vCPUs up to the max value, is done live and works well. But reducing the vCPUs requires a reboot.

      However. On a Rocky Linux 10 installation. The reducing works immediately live and online as expected.

      This might very well be a limitation of how Windows does things internally. But I thought it interesting enouch to report about anyway.

      posted in XCP-ng
      J
      jr-m4
    • RE: Why are transfer sizes different between XO5 & XO6?

      @JorisK said:

      fix/update-transfer-size-calculation

      I can confirm that XO6 now shows the same transfer sizes as XO5. Lovely

      posted in Backup
      J
      jr-m4
    • RE: Why are transfer sizes different between XO5 & XO6?

      @Danp
      I have Healtch Checks on my Delta Backups. Yes

      posted in Backup
      J
      jr-m4
    • RE: Why are transfer sizes different between XO5 & XO6?

      Same thing here:
      Commit: a1dba

      XO5 transfers ~ 9-20GB
      XO6 transfers ~ 100-180GB

      image.jpeg
      image.jpeg

      posted in Backup
      J
      jr-m4
    • RE: Troubleshooting "TCP: out of memory" - Possible memory leak?

      @florent & @poddingue

      I have now uploaded two heap-snapshots for you to look at.
      If there is anything else, big or small, that I can help out with. Just let me know

      Thanks!

      posted in Xen Orchestra
      J
      jr-m4