Thank you, low on sleep and of course you were right...

The problem is there's no such thing as "CPU architecture", that's only CPU features. It's a very complex problem and the easier way to solve it is to automatically harmonize to the lower one. Even if you are the oldest last, that will do as long as you shutdown/start previous VM again during a maintenance reboot of the guest OS.

You should have a command available in your Debian VM with tools, called "xenstore". You can use xenstore-write. For example: xenstore-write vm-data/gputemp 82 This will write the value 82 in a key gputemp. This key/value can be seen in the VM object then, eg with a : xe vm-param-get param-name=xenstore-data param-key=vm-data/gputemp uuid=<VM UUID> This will return 82. Now you can do whatever script in your Dom0 (or even from your XOA, since you can fetch all data in XAPI) As you can see, it's very simple and efficient

@AndyK I have had to sometimes do an iLO reset (reboot) to make it calm the fans down after installing. Its weird my gen 10s on ESXi never needed anything installed to prevent the fans from screaming but on these Gen 11s i tried various linux distro and xcp-ng and the fans were always loud without the daemon installed in the OS.

@olivierlambert Wow really nice unit makes me want to jump on those lighter fanless unit as i have lots of those 1L unit that works perfectly but has a small fans that i can hear if i really focus. I just like fanless unit to avoid dust getting in lol.

@Andrew I’m good with the BIOS setting for my NUC11 and 13. I wish Intel would fix ASPM, but still not on the latest BIOS. Just can’t run xcp-ng yet because of the older kernel and passthrough issue on the NUC13. It will probably come soon as 4.19 is out of support pretty soon. Was actually hoping there would be a alternate kernel for testing already.

@planedrop thank you for the input.

I quickly discussed with Xen developers and they are curious about your issue. Can you do a lspci -vvvv in your Dom0, first while you have still the option pci=realloc, and once without it, and see if we can spot any difference regarding your NIC. This might be helpful (you can provide both outputs at https://paste.vates.tech)

please advice is there a way to check that RAID1 read speed can be a bottleneck ? like 3 VMs are more than NVME Raid can give and that's why VMs are slow in responce

@manilx Got this from ChatGPT, so I think I got this. Would be great to just get these values from a cli command but I'm not up to the task The output from smartctl you provided contains information about the Wear Leveling Count attribute of an SSD. This attribute is used to indicate the wear and tear of the NAND flash memory cells in the SSD. Let's break down the relevant parts of the output: "Wear_Leveling_Count": This is the attribute being measured. "value": 98: This is the normalized value of the Wear Leveling Count. Normalized values typically start at 100 for new drives and decrease as the drive ages and wears out. "raw": { "string": "11", "value": 11 }: The raw value is the actual measurement from the SSD's controller. In this case, it is 11. Interpretation Normalized Value (98): This indicates the relative health of the SSD. A value of 98 means the SSD is still in good health, as it is close to the initial value of 100. Raw Value (11): This typically represents the number of cycles or the wear level. In this case, it likely indicates the number of times the cells have been erased and written. Wear Level SSDs generally have a threshold for when they are considered worn out, often around a normalized value of 10 or lower. With a normalized value of 98, your SSD is in very good health and shows minimal wear. The raw value of 11 suggests the SSD has gone through 11 cycles of wear leveling, which is quite low."

@olivierlambert Thank you, that's what I thought and why I was willing to buy the lower end CPU in order to get the project through. At the time going for the larger Gold would have doubled the cost of each server.

@Danp DANP THANK YOU! You are the man. Just click on that interface and tried to find documentation. Thank you for your help - I am sure I will have a few more on this..

@cunrun @jorge-gbs any init errors in dom0 /var/log/kern.log re. GIM driver? Also, if you search some topics here covering this specific GPU, there were mixed results booting dom0 with pci=realloc,assign-busses. Maybe it worth a try.

@sshughes My understanding of the system so far (but take that with a grain of salt, in real life I'm sort of an accountant who's only larp'ing as a sysadmin here): the system is not meant to 'run on a single host and when it dies we replace it'. It's meant to run on three identical hosts which never all die at the same time (except for some GBU hitting their building). Therefore there is no focus on 'let's reconnect new hardware to existing SR and fire up the VMs stored on that SR'. After planning accordingly, I can live with that. Planning accordingly in my case means: 'well, if my single host dies (running two or three isn't worth it for me), I'll take a day of down time to restore from backups.'

Yep, only one, cause one tesla onboard, no any other video devices (maybe it is a problem) [20:53 xcp ~]# lspci | grep -i nvidia 05:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev ff)

@andyhhp giving this a shot now.