Only updated one host in my pool so far, but it has my TrueNAS VM and network performance looks comparable

edit:

Actually quite an improvement in comparison to my results above

root@FILE001:~ # iperf -c 10.10.1.126
------------------------------------------------------------
Client connecting to 10.10.1.126, TCP port 5001
TCP window size: 80.8 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.125 port 35576 connected with 10.10.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  6.30 GBytes  5.36 Gbits/sec
root@FILE001:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.125 port 5001 connected with 10.10.1.126 port 54773
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  3.57 GBytes  3.06 Gbits/sec

Also done a VM reboot now and no passthrough issues

@stormi Only updated my test host so far, but all looks good to me.... no passthrough issues!!!

Will try and update my pool over the weekend and run some FreeBSD speed tests before and after (been waiting for that patch to make it out into the world).

@stormi A bit late to the party again (must try harder ) as I have been moving my rack and my test host was not set up and main pool down to running on 2 hosts..... taking one more offline would make Ceph very unhappy!!!....

Anyway, both test updates applied to my test host and I haven't managed to break anything yet!!! So looks good from my point of view.

@stormi Security updates on April 1st

Updating my pool now.... quite a substantial speed increase on FreeBSD

root@FILE001:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.125 port 5001 connected with 10.10.1.126 port 42952
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.1 sec   694 MBytes   576 Mbits/sec
^Croot@FILE001:~ # iperf -c 10.10.1.126
------------------------------------------------------------
Client connecting to 10.10.1.126, TCP port 5001
TCP window size:  105 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.125 port 50605 connected with 10.10.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   202 MBytes   169 Mbits/sec
root@FILE001:~ # iperf -c 10.10.1.126
------------------------------------------------------------
Client connecting to 10.10.1.126, TCP port 5001
TCP window size: 80.8 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.125 port 45072 connected with 10.10.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  4.06 GBytes  3.49 Gbits/sec
root@FILE001:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.125 port 5001 connected with 10.10.1.126 port 47101
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.17 GBytes  1.86 Gbits/sec
^Croot@FILE001:~ #

@stormi said in XCP-ng 8.2.0 RC now available!:

yum update uefistored

I could only get it (uefistored-0.2.6-1.xcpng8.2.x86_64) to update by yum update uefistored --enablerepo=xcp-ng-testing

But it has done the trick. No toolstak restart or reboot needed either.

@stormi Nice to do something useful and give a little back, other than buying a XCP-ng cushion for my office!

Just got this working in my 3 host home setup..... But I'm looking to drop down to two hosts. Is it going to be usable with 2 hosts (I've seen the recommendation of 3+ at the top) and if so, what happens when you get down to 1 host whatever reason??? Are read / writes locked on the remaining host?

if this helps....

yum install --enablerepo=epel -y qemu-dp xenopsd xenopsd-cli xenopsd-xc xcp-ng-xapi-storage runx
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile
 * centos-ceph-nautilus: mirror.as29550.net
 * centos-nfs-ganesha28: mirrors.vinters.com
 * epel: mirror.freethought-internet.co.uk
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-linstor: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-runx: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Package 2:qemu-dp-2.12.0-2.0.5.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-cli-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-xc-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xcp-ng-xapi-storage-1.0.2-3.0.0.runx.1.xcpng8.2.x86_64 already installed and latest version
Package runx-2021.1-1.0.0.runx.1.xcpng8.2.x86_64 already installed and latest version
Nothing to do

@olivierlambert ok, if its still planned to be a feature then I'm up for playing.... testing!!

@ronan-a is this the best way to get the template? The rest of the instructions look pretty simple to follow so I don't think I've got them wrong.....

xe vm-install template=Debian\ Buster\ 10 new-name-label=tempforrunx sr-uuid=7c5212f3-97b2-cdeb-b735-ad26638926e3 --minimal
this uuid is of the SR created by the step in the first post?

xe vm-param-set uuid=a2d46568-c9ab-7da2-57cb-d213ee9d8dfa HVM-boot-policy=""
uuid that is a result of the first step?

xe vm-param-set uuid=a2d46568-c9ab-7da2-57cb-d213ee9d8dfa PV-args=""
uuid that is a result of the first step?

xe vm-param-set VCPUs-max=2 uuid=14d91f2f-a103-da0e-51b3-21c8db307e5d
what is this uuid?

xe vm-param-set VCPUs-at-startup=2 uuid=14d91f2f-a103-da0e-51b3-21c8db307e5d
same again, where does this uuid come from?

xe vm-disk-remove device=0 uuid=cb5a6d67-07d5-b5ea-358a-7ee0d6e535af
and this one too?

xe template-param-set is-a-template=true uuid=a2d46568-c9ab-7da2-57cb-d213ee9d8dfa
uuid generated by the 1st step

Thanks.

@jjgg it would be great if we could get this working. My CPU utilisation is fine too, but when I shut down my Zoneminder VM things go a lot quieter (fans) so I'm sure there would be a benefit CPU and power wise.

Maybe this one will come to life again https://xcp-ng.org/forum/topic/7066/coral-tpu-pci-passthrough/14

Don't really want to buy one knowing its not working!!

Ah, just found these things exist..... Then just found this issue exists too ️

So no luck

lspci | grep USB
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

xl pci-assignable-list
0000:00:11.4
0000:04:00.0

vm.start
{
  "id": "da3eae3e-a67a-f0e2-f113-dd67f65baef1",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\"))"
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:63acb07f-292f-4062-820f-98ea4934b653",
      false,
      false
    ]
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\")))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\")))
    at Function.wrap (/home/node/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at /home/node/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:35:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.topLevelDomainCallback (domain.js:147:15)
    at process.callbackTrampoline (internal/async_hooks.js:129:24)"
} 

Any ideas anyone before I give up and send the card back......

EDIT

OK, computer says NO!!!! - This indicates that your device is using RMRR (opens new window). Intel IOMMU does not allow DMA to these devices (opens new window)and therefore PCI passthrough is not supported

EDIT 2

Could we patch the kernel too?

https://github.com/kiler129/relax-intel-rmrr/blob/master/README.md#other-distros

@jevan223 Thanks. So looks like yours is Renesas uPD720201 and mine is ASMedia -ASM1142. Might try another card then. Though I don't really want a collection lying around gathering dust oh well.

@jevan223 no nothing special for AMD cards. It just shows in XO as assignable. Used my WX4100 today but I was using a RX580 before.

I am using a PCIe card but no joy. I've got NICs and SAS & SATA controllers also passed through so I know that is all working. So maybe is my card that doesn't want to play. Could you link your specific card please?

I have passed through individual USB devices and that works nicely, but it's a bit of a faf to do.

Performance seams to be great. Just waiting for some LAN KVM devices to play with now!

Hi @jevan223 , I tried to do something very similar with an AMD card and USB card a few years ago. Got the AMD card working but couldn't get the USB card to pass-through. How did you manage that?

@scboley I would assume so, but I can't say yes. I don't think it was available before 8.2 without following the above.

@scboley https://xcp-ng.org/docs/storage.html#cephfs

Once you do the manual stuff it will show up like any other SR in Xen Orchestra etc.

if this helps....

yum install --enablerepo=epel -y qemu-dp xenopsd xenopsd-cli xenopsd-xc xcp-ng-xapi-storage runx
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile
 * centos-ceph-nautilus: mirror.as29550.net
 * centos-nfs-ganesha28: mirrors.vinters.com
 * epel: mirror.freethought-internet.co.uk
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-linstor: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-runx: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Package 2:qemu-dp-2.12.0-2.0.5.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-cli-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-xc-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xcp-ng-xapi-storage-1.0.2-3.0.0.runx.1.xcpng8.2.x86_64 already installed and latest version
Package runx-2021.1-1.0.0.runx.1.xcpng8.2.x86_64 already installed and latest version
Nothing to do