@Shrubbery said in xcp-ng 8.3 not recognizing all host memory:

The issue was memory reserved for the integrated graphics card

haha.. that is such a silly thing... but good to know then next time I'm working on some hardware and wondering where the RAM is going too...

@scawtydont Thanks for keeping us posted. Dummy plug is still a thing it seems 😄

If you are that much confident, then feel free to purchase one and use it: XCP-ng supports SRIOV but for NICs, I don't know how different it could be for a GPU, but without any hardware to test it's impossible to answer. How difficult it was to work on a more simpler hypervisor (KVM) doesn't send encouraging messages though.

@CodeMercenary Did some tests with the A2000 and as expected, the 12GB VRAM is the biggest limitation. Used vanilla installations of Ollama and ComfyUI with no tweaking or optimization. Especially in stable diffusion, the A2000 is about three times faster compared to the P40, but that is to be expected. I have added some results below.

Stable Diffusion tests

A2000 1024x1024, batch 1, iterations 30, cfg 4.0, euler 1.4s/it 1024x1024, batch 4, iterations 30, cfg 4.0, euler 2.5s/it = 0.6s/it P40 1024x1024, batch 1, iterations 30, cfg 4.0, euler 2.8s/it 1024x1024, batch 4, iterations 30, cfg 4.0, euler 12.1s/it = 3s/it

Inference tests

A2000 qwen2.5:14b 21 token/sec qwen2.5-coder:14b 21 token/sec llama3.2:3b-Q8 50 token/sec llama3.2:3b-Q4 60 token/sec P40 qwen2.5:14b 17 token/sec qwen2.5-coder:14b 17 token/sec llama3.2:3b-Q8 40 token/sec llama3.2:3b-Q4 48 token/sec

During heavy testing, the A2000 reached 70°C and the P40 reached 60°C both with the Dell R720 set to automatic fan control.

@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:

@ravenet Ok I think the pro gpu's work better without so many bugs.

I switched to Nvidia RTX 4070. With the Nvidia gpu and this works as it should and gets an image on the screen on the first try.

So there must be something wrong with the amd drivers plus that error with the amd reset bug

I've setup a test system with a ryzen 7700x and a regular radeon 7600xt, non-pro gpu. Will let you know results. Shouldn't be an issue, it's same hardware and even the drivers work across them. Though they do have different bios on cards

@Andrew Thanks Andrew sounds good , let me know when it suits..

@nick-lloyd thanks, I did! Passthrough for AMD consumer Hardare iGPUs seams not to properly function… or, I should say, the Linux VMs are experiencing troubles correctly dealing with it: the devices are correctly passing down to the VM, but for some reason the ROCm kernel module is not capable of dealing with them. Could it be be ause it expected them to be on a specific bus? (They are assigned a different bus id)
Could it be because it expected them to be under the same bus id? (They are scattered around different dedicated bus ids)
Not sure, but the end result is the iGPU cannot be correctly initialized, albeit detected.

@MasterOSkillio said in Intel CPUs with P and E cores or other server CPU suggestions:

From what I have seen in various places on the web P and E cores are not really supported yet for XCP-NG? Is it safe to assume that is changing? I believe proxmox, and VMware have moved to support them. If I use the 12600K will I have to turn off one (P or E) and use only the other?

Not sure what you mean by "support" but I'm running as of now the P+E core setup and it works just fine. Stable, performs as expected.

@gduperrey (unrelated to this fan issue) I loaded the new standard 8.2 testing kernel on my NUC11 and it seems to boot a little faster and also no longer complains about some APIC devices.

Regarding memory test: Just running the normal mem test from Grub should do, I guess?

Do we have a list of GPUs that actually do support vGPU (not Passthrough) in 8.3?

The documentation seems to be lagging behind there, a lot of references to the AMD mxgpu setups which have been deprecated.

@olivierlambert Correct.

When switching to the virtual console through iDRAC it works.

Is there a way to set the resolution of the OS when it starts and goes into xsconsole?

Same thing happens on the OS installer as well. Maybe the resolution is too low for the screen.

Ah indeed it makes sense then 🙂

Hi Teddy,
thanks for the analysis and your brief explanation.
IOMMU should be properly activated:

(XEN) [ 0.221843] AMD-Vi: IOMMU Extended Features:
(XEN) [ 0.222606] - Peripheral Page Service Request
(XEN) [ 0.223366] - NX bit
(XEN) [ 0.224123] - Guest APIC Physical Processor Interrupt
(XEN) [ 0.224889] - Invalidate All Command
(XEN) [ 0.225649] - Guest APIC
(XEN) [ 0.226412] - Performance Counters
(XEN) [ 0.227178] - Host Address Translation Size: 0x2
(XEN) [ 0.227940] - Guest Address Translation Size: 0
(XEN) [ 0.228681] - Guest CR3 Root Table Level: 0x1
(XEN) [ 0.229416] - Maximum PASID: 0xf
(XEN) [ 0.230140] - SMI Filter Register: 0x1
(XEN) [ 0.230867] - SMI Filter Register Count: 0x1
(XEN) [ 0.231596] - Guest Virtual APIC Modes: 0x1
(XEN) [ 0.232316] - Dual PPR Log: 0x2
(XEN) [ 0.233024] - Dual Event Log: 0x2
(XEN) [ 0.233727] - Secure ATS
(XEN) [ 0.234424] - User / Supervisor Page Protection
(XEN) [ 0.235126] - Device Table Segmentation: 0x3
(XEN) [ 0.235826] - PPR Log Overflow Early Warning
(XEN) [ 0.236514] - PPR Automatic Response
(XEN) [ 0.237198] - Memory Access Routing and Control: 0x1
(XEN) [ 0.237881] - Block StopMark Message
(XEN) [ 0.238558] - Performance Optimization
(XEN) [ 0.239234] - MSI Capability MMIO Access
(XEN) [ 0.239906] - Guest I/O Protection
(XEN) [ 0.240570] - Enhanced PPR Handling
(XEN) [ 0.241231] - Invalidate IOTLB Type
(XEN) [ 0.241886] - VM Table Size: 0x2
(XEN) [ 0.242537] - Guest Access Bit Update Disable
(XEN) [ 0.252988] AMD-Vi: IOMMU 0 Enabled.
(XEN) [ 0.253820] I/O virtualisation enabled

I have also read about the problem with UEFI several times.
Now I am going to try to find a system with "legacy" boot mode.

@knut99 Note for the UH125, it uses the RTL8126 ethernet chip. While it may work with XCP, it's officially not supported due to the Realtek drivers.

@imaginapix Didn't think of that.... But I guess it's more work than it's worth in this instance.

@manilx

True, not a big deal, but this two host were fresh installs with 8.3 release and only joint the pool to keep everything running.

So this seems to be a thing on pool level, since on those hosts only the Legrand was ever attached to.

I have been able to run a script at startup that enables 2.5g on the Intel X550. Here's how :

Create the script that will run at startup

nano /usr/local/bin/x550.sh

Content of x550.sh :

#!/usr/bin/bash sleep 1m ethtool -s eth1 advertise 0x1800000001028 echo `date +%F" "%T` "Startup script worked" >> /root/x550.log

Make the script executable :

chmod +x /usr/local/bin/x550.sh

Create the service that runs the script at startup

nano /etc/systemd/system/x550.service

Content of x550.service :

[Unit] Description=Runs /usr/local/bin/x550.sh [Service] ExecStart=/usr/local/bin/x550.sh [Install] WantedBy=multi-user.target

Enable the service

systemctl enable x550

Reboot and run this command to see if the link speed is 2.5g

ethtool ethX (replace X with yours)

It should show this result :

Supported ports: [ TP ] Supported link modes: 100baseT/Full 1000baseT/Full 10000baseT/Full 2500baseT/Full 5000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 100baseT/Full 1000baseT/Full 10000baseT/Full 2500baseT/Full 5000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 2500Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes

You can check the log file of your script located at /root/x550.log

I used this article to create everything :
https://www.redhat.com/sysadmin/replacing-rclocal-systemd

@olivierlambert Thank you - appreciate for the confirmation.