XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    • Profile
    • Following 0
    • Followers 1
    • Topics 2
    • Posts 123
    • Groups 4
    TeddyAstieT Offline
    1. Home
    2. TeddyAstie
    3. Posts

    Posts

    Recent Best Controversial
    • RE: Bringing container visibility back to XO

      While I think we need to do something about containers in VMs, I'm not convinced this is a good idea adding this to the regular guest agent for various reasons.

      Docker is one way of running containers, but there are numerous other ones like podman or other runc/containerd based ones. So we can't really make a one size fits all solution. And that doesn't fully answer the maintenance and long-term aspect.

      Well, actually what you are looking for is some form of control on guest docker runtime, but without using network, that's actually a solved problem, but plumbing is missing. And we don't need a custom docker-aware agent for this, we mostly need to use one of vsock/pv-channel/pvcalls to build a bridge between e.g guest docker socket and Xen Orchestra (which would talk to guest docker and control/get info from it).

      posted in Xen Orchestra
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @dkidd255 @jamesg
      I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control).

      In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: Migrating an offline VM disk between two local SRs is slow

      I'm not sure disabling Nagle is a good idea (even though it can improve things here). Fundamentally, we're doing bulk transfer of disk content, which Nagles tries to optimize by coalescing packets, so you're not flooding the network with small TCP packets.

      The main problem here is that the progress is gated by NBD replies, which is going to be bad regardless of TCP configuration. TCP_NODELAY will workaround this problem, but with significant tradeoffs (and perhaps will perform worse in some other cases).
      What should be done instead is that writes should be streamed (or pipelined) while reading replies in parralel, so that NBD reply delays doesn't bottleneck the whole transfer. But that actually requires a redesign of the whole NBD implementation which is not going to be a easy thing AFAICT.

      posted in Xen Orchestra
      TeddyAstieT
      TeddyAstie
    • RE: XCP-NG 9.0 Support for GRAID Tech GPU-accelerated RAID cards.

      I can't speak for the whole team, but will still give my view on it, at least on what I'm aware of.

      @hydromike a dit:

      ...

      VROC is the easier half

      Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter.

      VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0.

      Roughly what that path needs:

      • CONFIG_VMD in the dom0 kernel
      • The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30)
      • mdadm with IMSM in dom0
      • Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help

      Worth reminding that installing custom software in Dom0, or replacing Dom0 packages is discouraged.

      The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one.

      SupremeRAID is harder

      I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back.

      P2P DMA is more of a platform capability (e.g ACS stuff) than a hypervisor issue. At least, as long as there is no DMA to magic regions like MSI-X/PBA as that will potentially fail.

      In the past, I had success doing P2P DMA between a PCI card and NVMe drives, so it's definetely doable.

      ...

      Questions for @yann and @Team-Hypervisor-Kernel

      These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after:

      1. Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask.

      Dom0 will still be running in PV mode by default at least. There are no immediate plan to switch to PVH Dom0.

      1. What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere.
      2. What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing.
      3. Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM?

      Features like SecureBoot will add complexity with external out-of-tree modules, and while there can be solutions to that, but I don't expect any of that to land anytime soon.

      1. Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in?

      If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead.

      ...

      AIA, I don't expect anything practical with the current state of things unless there is a quite significant effort.
      However, if these stack(s) can be moved in a dedicated VM with PCI Passthrough, that can make things more workable with the likes of storage domains.

      posted in Development
      TeddyAstieT
      TeddyAstie
    • RE: i915 pass-through and Linux Mint - xcp-ng 8.3

      Intel needs some special handling to support physical displays with PCI Passthrough; I don't know much of the details, but on "recent" machines, some bits are missing according to :
      https://lore.kernel.org/all/20260802050824.10554-1-brchuckz@aol.com/

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @JamesG Don't bother, I'm in PTO next week; so I may get access to hardware when I'm back.

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      FYI I will get my hands on a Intel Arc Pro B50 to make various tests with it (in the next weeks). If something don't behave as expected, I will try to find a solution.

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @JamesG

      You can't PCI Passthrough a PCI bridge (and even if you could, it wouldn't be very useful), as it's a part of PCI internals and not really something the guest requires to use the device.

      As I asked previously, is there any issue (aside display like PCIe 1.0 1x) with the device, is it behavior correctly or having the expected performance ?

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: RDNA 4 GPU Passthrough

      @ravenet said:
      @teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

      TL;DR This is likely a known issue, and we're currently discussing on a potential fix.

      There is a known bug regarding MSI on AMDGPU when using "hvm_pirq" ("use event channels for device interrupts"), see
      https://lore.kernel.org/xen-devel/20240110095311.61809-1-roger.pau@citrix.com/
      https://github.com/QubesOS/qubes-issues/issues/7971

      The easy fix is to disable "HVM PIRQ" feature (which has been made disabled by default in Xen 4.19), but on the XCP-ng side, this is configured by XAPI Project which needs to be updated as well to apply this change (which appears to be a bit tricky for various reasons).

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: RDNA 4 GPU Passthrough

      @PessimistTech
      That looks a bit odd indeed.

      In addition to what proposed @yannsionneau, can you also give the output of :

      • xl dmesg (in Dom0)
      • dmesg (in Dom0)
      • dmesg (in the guest)

      So that we can try to pin-point what may be happening.

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @JamesG said:

      So I ordered some mini DP dummy plugs and took a look at this further today.

      First thought...Just load Windows on the bare metal and see what the card does.

      No problem. Loaded Win10, loaded the latest drivers, PCIe Gen4x8 lanes. Sweet. At least that works.

      Put the XCP-ng drive back in, booted up, passed-through the GPU to a Win11 guest, updated the drivers, PCIe Gen1x1.

      As I explained briefly in https://xcp-ng.org/forum/post/106479, this is mostly display as the real hardware may be running at a different speed.

      Ultimately I need to install a Linux distro with a really current kernel on the bare metal and see how it sees the GPU. That might be a good project for tomorrow.

      I think this just comes down to a combination of too new, not well supported hardware with some quirks and somewhat dated code and support in XCP-ng (ReBAR support in guests for example).

      I did quick checks and I believe that ReBAR works; at least with UEFI guests (our OVMF build does support resizable bar), and it actually works as suggest https://xcp-ng.org/forum/post/106489 (VRAM BAR is 16 GB); but I didn't do extensive testing in that area.

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @JamesG Can you try having a workload on the GPU (and also checking if the performance is lower, or if it's just display) ?

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @JamesG said:

      When I ask the system what it thinks the capabilities are, again I get Gen1x1:

      cat /sys/bus/pci/devices/0000:00:08.0/current_link_speed
      2.5 GT/s PCIe
      cat /sys/bus/pci/devices/0000:00:08.0/current_link_width
      1
      cat /sys/bus/pci/devices/0000:00:08.0/max_link_speed
      2.5 GT/s PCIe
      cat /sys/bus/pci/devices/0000:00:08.0/max_link_width
      1

      This information is actually crafted by QEMU, it doesn't represent the actual real device PCI state; you should check this from Dom0 which should be closer to reality.

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: PCIe Pass-through lanes and lane performance

      @JamesG

      I know I'm a bit out in uncharted territory here. I don't think XCP-ng supports ReBAR via pass-through (I think that's on the road-map for XenServer9 and maybe XCP-ng 9 as well), current gen GPUs all want ReBAR support, Intel GPU's are in a mostly experimental/developmental mode and might be having a murky future anyway...

      XCP-ng guest firmware have a very "legacy" initial BAR placement (i.e thus no explicit support for Resizable BAR, everything below 4G, ...), but the guest operating system can still replace the location of BAR itself if the hardware allows it (in particular Linux will try it, but I'm not sure Windows does).

      In your case, the GPU VRAM BAR is 16 GB, so it's obviously resized. And according to xe, you should see in dmesg something like BAR2 resized to 16384MiB

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: XCP-ng 8.3 updates announcements and testing

      @Andrew said:
      HP DL G8 Intel E5-2673 v2 shows 64 CPUs. The actual CPUs show correctly, the higher ones (that don't exist) show [CPUxx] Unable to fetch temperature (19 - No such device)

      Machines with hotpluggable CPUs are a pretty tricky case, the logic tries up to "maximum possible CPU" and fails here because the CPU is not online (No such device error). That doesn't prevent the temperature from getting fetched for CPU that exists.
      I can try to add a check to hide this specific error, so it's doesn't create noise for offline CPUs on such machines.

      posted in News
      TeddyAstieT
      TeddyAstie
    • RE: XCP-ng 8.3 updates announcements and testing

      @acebmxer said:

      @rzr

      Just installed updates on host 1. Once host rebooted it took an extra min or two to reconnect to xo, but did finally connect. Applying updates on host 2 now.

      Update - host2 no issues. Once reboot complete it connected to xo as expected without delay.

      I see these updates include -

      • xen: Add support for xenpm get-core-temp to query CPU temperature on Intel platforms.

      • Use xenpm get-core-temp to get the temperature on Intel's CPU, to fallback unsupported coretemp. Doc update being reviewed .

      My host are AMD so can verify these. I might be able to deploy a Intel host later tonight. Will this come to AMD later?

      AMD rely on a different method to expose the temperature, that don't require this xenpm-based approach. In principle, it should already work with plain sensors (through k10temp), but our driver may not be up to date for recent AMD CPUs.

      posted in News
      TeddyAstieT
      TeddyAstie
    • RE: Slow boot on rocky linux 10 latest kernel

      Regarding upstream Linux, it should be addressed with https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=f24df84cbe05e4471c04ac4b921fc0340bbc7752

      Although, I have no ETA on when it will land to distros.

      posted in Compute
      TeddyAstieT
      TeddyAstie
    • RE: Xen 8.2 isos

      @TrapoSAMA https://updates.xcp-ng.org/isos/8.2/
      And https://updates.xcp-ng.org/8/8.2/ appears fine.

      posted in Off topic
      TeddyAstieT
      TeddyAstie
    • RE: The Lowest Priority Bug Ever? (/etc/udev/rules.d/z10-xen-vcpu-hotplug.rules)

      The rule is oddly written, and may conflict with another similar one that already exist in the distro (hence may not be useful to begin with).

      The modern generic rule for doing vCPU hotplug is, which would be preferable to the current z10-xen-vcpu-hotplug.rules.
      ACTION=="add", SUBSYSTEM=="cpu", ATTR{online}=="0", ATTR{online}="1"

      posted in XCP-ng
      TeddyAstieT
      TeddyAstie
    • RE: Slow boot on rocky linux 10 latest kernel

      @MajorP93 said:
      Do you think it is possible to fix this on hypervisor level while still having live migration etc. enabled or do we have to wait for an upstream fix within Linux kernel tree?

      Yes it's possible to fix it on the hypervisor level (Invariant TSC in guest), but it's quite a bit of work that still needs to be done. A Linux upstream fix for the underlying bug should come at some point hopefully.

      posted in Compute
      TeddyAstieT
      TeddyAstie