XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    • Profile
    • Following 2
    • Followers 0
    • Topics 11
    • Shares 0
    • Posts 128
    • Groups 0
    C Offline
    1. Home
    2. cg

    cg

    @cg

    54
    Reputation
    56
    Profile views
    128
    Posts
    0
    Followers
    2
    Following
    Joined
    Last Online

    cg Unfollow Follow
    • RE: XCP-ng 8.0.0 Beta now available!

      @stormi said in XCP-ng 8.0.0 Beta now available!:

      You could test with Citrix Hypervisor and then report it at https://bugs.xenserver.org
      Maybe they'll want to investigate even if the hardware is not officially supported. If someone can reproduce the issue on supported hardware, that would make things easier.

      Though I doubt it was woth the effort, as the support over there is well known (especially for not supported configurations etc.) I did it - for the reason of trying to improve stuff (and giving back to OSS community):
      https://bugs.xenserver.org/projects/XSO/issues/XSO-955

      posted in News
      C
      cg
    • RE: Citrix Hypervisor 8.0 landed

      @stormi dd stands for disk dump and does exactly that: Copy a stream of data.
      Fio however can be configured for precise workloads and read/write mixes, parallel workloads etc.

      So the first thing will only give you streamline benchmarks, what almost nobody cares about.
      The second can simulate realworld (VM/database...) workloads, where (controller) Caches and non magnetic storage (Flash, Optane, MRAM...) makes the real difference.
      Also use big amount of data, since caches can impact small ones extremely. Don't get me wrong: We need them and they can make huge differences, but as long as your benchmarks fully fit into them, it gives your nonsense/fake results. Also (consumer) SSDs start throttling after some 10 to a very few 100 GB of data written. Their caches fill up and they 'overheat'.

      You can spend days on benchmarks and how to do what. 😉

      posted in News
      C
      cg
    • RE: First SMAPIv3 driver is available in preview

      @olivierlambert said in First SMAPIv3 driver is available in preview:

      Hi,

      8.3 release is eating a lot of resources, so that's the opposite: when it's out, this will leave more time to move forward on SMAPIv3 🙂

      Lots of work means lots of changes, means: I'm exciting about it. Also sounds more like a 9.0, if that much work is going into it. 😉

      posted in Development
      C
      cg
    • RE: VM console borks with "Alt Gr" key

      The problem is still persistent or exisiting again.

      Exactly: You can use ALT GR once and get the symbol you need (@ |...) but afterwarts it's just putting weird symbols into the console.
      Dirty fix: Change the tab to whatever (general, stats...) and and back to console.

      Still pretty annoying and kicked me out of one of my servers for some time (thx fail2ban...). 😕

      posted in Xen Orchestra
      C
      cg
    • Blog Entry for 5.36

      https://xen-orchestra.com/blog/xen-orchestra-5-36/

      The part about ZFS:

      As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create your SR.

      Check this documentation for more information about ZFS SR creation.

      Once the SR is created, you will have the option to select it during the SR creation directly in Xen Orchestra.

      Since it's confusing and technically wrong, it should rather be something like:

      As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create a ZFS pool before creating the corresponding SR.

      Check this documentation for more information about ZFS SR creation.

      Once the ZFS pool is created and mounted, you will have the option to select the mountpoint during the SR creation directly in Xen Orchestra.

      ...and this forum is missing a "quote" button, as formating text in codeboxes doesn't work (and tags don't survive a linebreak, you have to mark it every line again).

      Regards,
      Christof

      posted in Xen Orchestra
      C
      cg
    • RE: Upgrade 8.2.1 -> 8.3 failed (manually fixed)

      @bleader

      IIRC I just "tried again".
      It failed 2 times, then I looked up the logs from other console, removed the file (which shouldn't be of any importance for our instance) and retried without reboot.

      I copied the whole installer-log to the usb stick before finshing the install. 🙂
      (Could actually be a good hint or even a menu-option for those, where the install fails and won't leave it on the harddrive - e.g. evaluating hardware)

      [  128.517356] ata1.00: exception Emask 0x0 SAct 0x800000 SErr 0x0 action 0x0
      [  128.517357] ata1.00: irq_stat 0x40000008
      [  128.517359] ata1.00: failed command: READ FPDMA QUEUED
      [  128.517362] ata1.00: cmd 60/80:b8:10:6c:d4/00:00:02:00:00/40 tag 23 ncq dma 65536 in
               res 41/40:10:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
      [  128.517363] ata1.00: status: { DRDY ERR }
      [  128.517364] ata1.00: error: { UNC }
      [  128.518008] ata1.00: configured for UDMA/133
      [  128.518018] sd 0:0:0:0: [sda] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
      [  128.518020] sd 0:0:0:0: [sda] tag#23 Sense Key : Medium Error [current] 
      [  128.518021] sd 0:0:0:0: [sda] tag#23 Add. Sense: Unrecovered read error - auto reallocate failed
      [  128.518024] sd 0:0:0:0: [sda] tag#23 CDB: Read(10) 28 00 02 d4 6c 10 00 00 80 00
      [  128.518025] print_req_error: I/O error, dev sda, sector 47475840
      [  128.518039] ata1: EH complete
      [  128.581286] ata1.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x0
      [  128.581287] ata1.00: irq_stat 0x40000008
      [  128.581288] ata1.00: failed command: READ FPDMA QUEUED
      [  128.581291] ata1.00: cmd 60/08:c8:80:6c:d4/00:00:02:00:00/40 tag 25 ncq dma 4096 in
               res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
      [  128.581292] ata1.00: status: { DRDY ERR }
      [  128.581293] ata1.00: error: { UNC }
      [  128.582111] ata1.00: configured for UDMA/133
      [  128.582117] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
      [  128.582118] sd 0:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] 
      [  128.582119] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed
      [  128.582121] sd 0:0:0:0: [sda] tag#25 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
      [  128.582122] print_req_error: I/O error, dev sda, sector 47475840
      [  128.582133] ata1: EH complete
      [  128.629307] ata1.00: exception Emask 0x0 SAct 0x200 SErr 0x0 action 0x0
      [  128.629309] ata1.00: irq_stat 0x40000008
      [  128.629310] ata1.00: failed command: READ FPDMA QUEUED
      [  128.629313] ata1.00: cmd 60/08:48:80:6c:d4/00:00:02:00:00/40 tag 9 ncq dma 4096 in
               res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
      [  128.629314] ata1.00: status: { DRDY ERR }
      [  128.629315] ata1.00: error: { UNC }
      [  128.630068] ata1.00: configured for UDMA/133
      [  128.630074] sd 0:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
      [  128.630076] sd 0:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] 
      [  128.630077] sd 0:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
      [  128.630078] sd 0:0:0:0: [sda] tag#9 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
      [  128.630079] print_req_error: I/O error, dev sda, sector 47475840
      [  128.630092] ata1: EH complete
      

      Indeed it looks like the SSD should be replaced.

      8.3 is running stable on this (and all other hosts, I upgraded) so far.
      It's a system at a UAS, running various student projects for several years now, coming from XenServer originally. I voluntarily maintain it. Thx for the hint!

      posted in XCP-ng
      C
      cg
    • RE: Windows 2025 Standard 24H2.11 (iso release of sept 25) crash on reboot with "INACCESSIBLE BOOT DEVICE 0x7B" in XCP 8.2.1 and XCP 8.3

      @dinhngtu I can't say on XCP-ng side, but it's likely linked to:
      August patch (and following), as Microsoft changed something to the NVMe stack.

      e.g.
      https://learn.microsoft.com/en-us/answers/questions/5536733/potential-ssd-detection-bug-in-windows-11-24h2-fol

      Google gives a lot about it. It seems that it most likely doesn't kill NVMes but can cause trouble.
      We have a few PCs becoming more unstable (BSODs) or even very slow after that upgrae.

      posted in XCP-ng
      C
      cg
    • RE: Bug in XO-Lite GUI / VM Creation

      @poddingue Yes, reads pretty much like the same error / root cause.

      posted in XO Lite
      C
      cg
    • RE: Backup solutions for XCP-ng

      @olivierlambert
      Okay, so bascially what I (and other people, I know, who are in a similar environment) need:

      • Tape backup is mandatory, as we're talking about (deduplicated, compressed) dozens of TB
      • Deduplication and good compression: duplicated, uncompressed ends in PB
      • All of that needs to scale! E.g. Commvault scales on 16 threads here even with only 1 - 2 tasks. (Can't say if even more without upgrade of CPU)
        (We need chains, of course, to go B2D2T -> dumping the dedup-store on tape as disaster recovery)
      • Applicationawareness (which can be done via agent - agentless is not always king or very important):
        -- MS Exchange, recovering datastores, mailboxes and even single mail items
        -- SQL Servers: MS SQL, MySQL (MariaDB)...
        -- Windows Active Directory Items

      Over here I also need to have the option to do backups only inside the VM via agent, as I can't snapshot them.
      You might have gotton me wrong: AFAIK quiesced backups always made a VM snapshot and submitted the request also to the VSS writer and then the VM/VHDs have been "read". In this case it will always fail, as there's no space on storage for such. Also Citrix discontinued that in their VM tools! XCP-ng yet lacks a proper tool maintenance with corresponding releases, which would be the only chance to keep that.

      As an idea for XOA:

      • Make it run on physical hardware, as VMs are too small for such
      • You already cooperate with HPE - let them run bare metal on ProLiants!
      • ZFS(oL) Could be your way to go to implement dedup and compression

      ...feel free to add, comment... whatever.

      posted in News
      C
      cg
    • RE: XCP-ng 8.2 updates announcements and testing

      Citrix also has a long history of problematic VM guest-tools. Race conditions with broken drivers, unbootable VMs, broken agents... It's a pain in the a**.
      If you like your VMs: Always make a snapshot of every VM before updating the tools.

      Is known why they remove the VSS-functionality? I don't get why they remove features (again).

      posted in News
      C
      cg
    • RE: Booting Homeassistant (HAOS) Image breaks VM

      @IgorGlock What do you mean by "okay"?
      Backing up the VM with Commvault or...?

      posted in Compute
      C
      cg
    • RE: Booting Homeassistant (HAOS) Image breaks VM

      Hey Poddingue!

      I very much assume to have found the troublemaker.
      I compared the outputs of xe vm-param-list and what triggered my eyes the most was:

      Debian:

      PV-drivers-version (MRO):
      

      HA:

      PV-drivers-version (MRO): major: @PRODUCT_MAJOR_VERSION@; minor: @PRODUCT_MINOR_VERSION@; micro: @PRODUCT_MICRO_VERSION@; build: @NUMERIC_BUILD_NUMBER@
      

      Ofc what's reported after a HomeAssistant boot is completely nuts and should be "nothing"/empty.
      As the errorlog of Commvault says something about XenSDKWrapper, I'm unsure if the errorhandling of the SDK or Commault sucks (should I be happy that it was caught at all and didn't crash the whole application? 🙄 ). The German translation, to me, indicateds Commvault at faul though. Neither XCP-ng nor the XenSDK will be able to speak German, nor is it likely that they translate a "try-catch"-error message, I assume.

      BUT: XCP-ng should never report back such nonsense and by that probably hit Commvault unexpected. So I also suggest to catch nonsense/unclear responses before XAPI (which is what xe triggers?) output/response.

      posted in Compute
      C
      cg
    • RE: Bug in XO-Lite GUI / VM Creation

      @poddingue Yes, reads pretty much like the same error / root cause.

      posted in XO Lite
      C
      cg
    • Bug in XO-Lite GUI / VM Creation

      For a while now, there's a bug in the VM creation dialog:
      grafik.jpeg

      If you want to change the disk name, it drops focus after 1 single character. You have to re-focus on it again (clicking the box - and watching it doesn't mark everything inside and... replaces it), enter one more character... do loop until...

      That error persists for months now.
      I've only noticed that in sotrage / disk name field, nowhere else.

      Regards

      • Christof
      posted in XO Lite
      C
      cg
    • RE: 9.2.385 snapshot/backup corruption fix: which versions were affected, and does it apply to XenServer VM Tools?

      @trelane said:
      When I turned off "Manage Citrix PV drivers via Windows Update" (has-vendor-device=false) to switch from XenServer VM Tools, Windows treated the NICs as new adapters.

      That bug had already cost me so much time. It has a looong history and seems to be "nailed in" when the VM had old tools installed at some point.
      I didn't have that problem with newer VMs that never had old tools tho.

      posted in Compute
      C
      cg
    • Booting Homeassistant (HAOS) Image breaks VM

      Hello!

      I have a very weird problem/bug!

      A while ago I imported the Homeassistant VMDK via XOA and stuffed it into an (Generic Linux UEFI) VM.
      It booted and runs flawlessly. Everything works - except backup via Commvault!
      Commvault fails reporting the VM could not be found. Support found the following inside the logfiles:

      4116  1984  05/01 21:00:29 58533 VSDisc::ProcessContent_VM() - virtual machine is not found using Name. Searching using both Name [HomeAssistant] and GUID [c3e276ed-1922-08a5-7f95-8b2d630e84ff]
      4116  1984  05/01 21:00:29 58533 CXenSDKWrapper::GetVMInfoForVM() - Error:Getting VM information failed.Die Eingabezeichenfolge hat das falsche Format.
      4116  1984  05/01 21:00:29 58533 VSDisc::ProcessContent_VM() - virtual machine: [c3e276ed-1922-08a5-7f95-8b2d630e84ff] discovery failed
      

      I assume "Die Eingabezeichenfolge hat das falsche Format." would be "The (input) string has the wrong format" in english.

      What I can: Discover the pool and select the VM to be taken into a backup job. So it somehow can see it, but it can't when running the actual job.
      Here's where things go really weird:
      I've spend quite some time, trying to narrow it down to a root cause. I've tried different VM templatesm, changed names and removed characters etc.
      Then I've interchanged the VDI with another VMs. Surprise:
      When I've interchanged HAOS VDI with a Debian Linux VMs OS VDI, the VM that cannot be backed up follows HAOS.
      Means: Now The VM named "Homeassistant" is found and backed up, but the VM called Debian-Test is not.
      So it turns out: Whenever a VM has booted HAOS, it can't be found during backup-discovery anymore. If it booted a "standard" Debian, the same VM is found again - until I change back to HAOS VDI.
      The support was clueless and we decided that I should try it over here: It looks like HAOS has to change something at the VM "skeleton" / metadata, that it can't be found otherwise.

      But how do I debug that further? How to find the difference and what exactly happens when it boots/runs?

      Greetings

      • Christof
      posted in Compute
      C
      cg
    • RE: 9.2.385 snapshot/backup corruption fix: which versions were affected, and does it apply to XenServer VM Tools?

      @trelane Seems to be a pretty annoying task. It says Upgrade from 9.2.350 is blocked, Need to apply "9.2.351 compatibility update" first. 😕

      I just read the release notes which are buggy:
      Important note for users upgrading from 9.2.350: You must first install the included XenTools-fix-9.2.351.msp hotfix to upgrade to 9.2.351 before installing 9.2.385.

      The zip-file does not containe that msp!
      ...it's a separate file to download.

      posted in Compute
      C
      cg
    • RE: USB-Passthrough does not survive reboot of VM

      @DustinB doesn't it use the exact same mechanism?
      I have to find out.

      posted in XCP-ng
      C
      cg
    • USB-Passthrough does not survive reboot of VM

      I had that many times with XCP-ng 8.2 and it felt like the problem is gone with 8.3, but it seems it's not (completely) gone.

      I can enable passthrough, add it to VM and it runs fine.
      I shutdown the VM for whatever reasons, recently it was just a reboot for Windows updates and the USB device got lost.

      What could be the reason, how to debug that?

      6487a635-933d-49ec-8ace-8a2b40e0a2f3-image.jpeg

      I can enable it again, shutdown VM (hotplug sadly still doesn't work till today?) attach and it's fine again.

      Regards

      • Christof
      posted in XCP-ng
      C
      cg
    • RE: Question on CPU masking with qemu and xen

      As I can't find it anywhere:
      What exactly is happening on join/leave?
      Is the masking changing at runtime and affecting every new started VM, does the pool need to be rebooted after an older server has been removed, to "unlock" all features?

      I have a pool with HPE DL380 Gen 9 and 10 where Gen 9 will replaced with Gen 12.

      So I assume it makes sense to remove 9 first, then add 12.
      But how exactly is the masking process working?
      Should I reboot Gen 10 first (shutting down all VMs) or add 12, move VMs over... or just shutdown and start all currently running VMs?

      In the early days (~XenServer 6) it had to be done manually by getting CPU features, calculating the commons and give it as parameter to join command.
      What I did not see though, is: How exactly is it working on removal and what's best practice.

      (also: should be added to documentations)

      posted in Compute
      C
      cg