@jjgg it would be great if we could get this working. My CPU utilisation is fine too, but when I shut down my Zoneminder VM things go a lot quieter (fans) so I'm sure there would be a benefit CPU and power wise.

Maybe this one will come to life again https://xcp-ng.org/forum/topic/7066/coral-tpu-pci-passthrough/14

Don't really want to buy one knowing its not working!!

Ah, just found these things exist..... Then just found this issue exists too ️

So no luck

lspci | grep USB
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

xl pci-assignable-list
0000:00:11.4
0000:04:00.0

vm.start
{
  "id": "da3eae3e-a67a-f0e2-f113-dd67f65baef1",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\"))"
  ],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:63acb07f-292f-4062-820f-98ea4934b653",
      false,
      false
    ]
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\")))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Xenctrlext.Unix_error(30, \"1: Operation not permitted\")))
    at Function.wrap (/home/node/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at /home/node/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:35:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/home/node/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/home/node/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/home/node/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.topLevelDomainCallback (domain.js:147:15)
    at process.callbackTrampoline (internal/async_hooks.js:129:24)"
} 

Any ideas anyone before I give up and send the card back......

EDIT

OK, computer says NO!!!! - This indicates that your device is using RMRR (opens new window). Intel IOMMU does not allow DMA to these devices (opens new window)and therefore PCI passthrough is not supported

EDIT 2

Could we patch the kernel too?

https://github.com/kiler129/relax-intel-rmrr/blob/master/README.md#other-distros

@jevan223 Thanks. So looks like yours is Renesas uPD720201 and mine is ASMedia -ASM1142. Might try another card then. Though I don't really want a collection lying around gathering dust oh well.

@jevan223 no nothing special for AMD cards. It just shows in XO as assignable. Used my WX4100 today but I was using a RX580 before.

I am using a PCIe card but no joy. I've got NICs and SAS & SATA controllers also passed through so I know that is all working. So maybe is my card that doesn't want to play. Could you link your specific card please?

I have passed through individual USB devices and that works nicely, but it's a bit of a faf to do.

Performance seams to be great. Just waiting for some LAN KVM devices to play with now!

Hi @jevan223 , I tried to do something very similar with an AMD card and USB card a few years ago. Got the AMD card working but couldn't get the USB card to pass-through. How did you manage that?

@scboley I would assume so, but I can't say yes. I don't think it was available before 8.2 without following the above.

@scboley https://xcp-ng.org/docs/storage.html#cephfs

Once you do the manual stuff it will show up like any other SR in Xen Orchestra etc.

if this helps....

yum install --enablerepo=epel -y qemu-dp xenopsd xenopsd-cli xenopsd-xc xcp-ng-xapi-storage runx
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile
 * centos-ceph-nautilus: mirror.as29550.net
 * centos-nfs-ganesha28: mirrors.vinters.com
 * epel: mirror.freethought-internet.co.uk
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-linstor: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-runx: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Package 2:qemu-dp-2.12.0-2.0.5.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-cli-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xenopsd-xc-0.150.12-1.2.xcpng8.2.x86_64 already installed and latest version
Package xcp-ng-xapi-storage-1.0.2-3.0.0.runx.1.xcpng8.2.x86_64 already installed and latest version
Package runx-2021.1-1.0.0.runx.1.xcpng8.2.x86_64 already installed and latest version
Nothing to do

@ronan-a thanks (was obviously too late for me to think about trying that!!) but no luck

[18:36 bad-XCP-ng-Host-03 ~]# xe vm-install template=Debian\ Buster\ 10 new-name-label=RunX sr-uuid=cb817299-f3ee-9d4e-dd5d-9edad6e55ed0 --minimal
b3b5efcf-e810-57ab-5482-4dba14dda0a6
[18:37 bad-XCP-ng-Host-03 ~]# xe vm-param-set uuid=b3b5efcf-e810-57ab-5482-4dba14dda0a6 HVM-boot-policy=""
[18:38 bad-XCP-ng-Host-03 ~]# xe vm-param-set uuid=b3b5efcf-e810-57ab-5482-4dba14dda0a6 PV-args=""
[18:38 bad-XCP-ng-Host-03 ~]# xe vm-param-set uuid=b3b5efcf-e810-57ab-5482-4dba14dda0a6 VCPUs-max=2
[18:38 bad-XCP-ng-Host-03 ~]# xe vm-param-set uuid=b3b5efcf-e810-57ab-5482-4dba14dda0a6 VCPUs-at-startup=2
[18:39 bad-XCP-ng-Host-03 ~]# xe vm-disk-remove device=0 uuid=b3b5efcf-e810-57ab-5482-4dba14dda0a6
[18:39 bad-XCP-ng-Host-03 ~]# xe template-param-set is-a-template=true uuid=b3b5efcf-e810-57ab-5482-4dba14dda0a6
[18:41 bad-XCP-ng-Host-03 ~]# nano /etc/runx.conf
[18:43 bad-XCP-ng-Host-03 ~]# podman container create --name archlinux archlinux
0f72916b8cca6c7c4aa68e6ebee467c0480e29e22ebbd6ec666d8385826f88b8
[18:43 bad-XCP-ng-Host-03 ~]# podman start archlinux
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: xenopsd internal error: Could not find BlockDevice, File, or Nbd implementation: {"implementations":[["XenDisk",{"backend_type":"9pfs","extra":{},"params":"vdi:1e2e7b99-2ffe-4129-a468-96b57ab685de share_dir none ///root/runx-sr/2"}]]}
archlinux6
[18:44 bad-XCP-ng-Host-03 ~]# rpm -qa | grep xenops
xenopsd-xc-0.150.12-1.2.xcpng8.2.x86_64
xenopsd-cli-0.150.12-1.2.xcpng8.2.x86_64
xenopsd-0.150.12-1.2.xcpng8.2.x86_64

Not sure what I'm doing wrong now, or if I need an update??

When I created the SR I had to give a host-uuid. I obviously used the host that I'm installing on.

@olivierlambert ok, if its still planned to be a feature then I'm up for playing.... testing!!

@ronan-a is this the best way to get the template? The rest of the instructions look pretty simple to follow so I don't think I've got them wrong.....

xe vm-install template=Debian\ Buster\ 10 new-name-label=tempforrunx sr-uuid=7c5212f3-97b2-cdeb-b735-ad26638926e3 --minimal
this uuid is of the SR created by the step in the first post?

xe vm-param-set uuid=a2d46568-c9ab-7da2-57cb-d213ee9d8dfa HVM-boot-policy=""
uuid that is a result of the first step?

xe vm-param-set uuid=a2d46568-c9ab-7da2-57cb-d213ee9d8dfa PV-args=""
uuid that is a result of the first step?

xe vm-param-set VCPUs-max=2 uuid=14d91f2f-a103-da0e-51b3-21c8db307e5d
what is this uuid?

xe vm-param-set VCPUs-at-startup=2 uuid=14d91f2f-a103-da0e-51b3-21c8db307e5d
same again, where does this uuid come from?

xe vm-disk-remove device=0 uuid=cb5a6d67-07d5-b5ea-358a-7ee0d6e535af
and this one too?

xe template-param-set is-a-template=true uuid=a2d46568-c9ab-7da2-57cb-d213ee9d8dfa
uuid generated by the 1st step

Thanks.

@olivierlambert Thanks. I tried to find a definite answer in the Linstor docs but couldn't.

I had set replication = 2 and could see the diskless copies in the volume list.

I then shutdown one host, and then saw the auto eviction after an hour and overnight it has adjusted so each of the two remaining hosts has a complete replica, i.e. there are no more diskless copies. So it looks like a nice easy to deploy solution (if you need the CPU power of 3 hosts rather than just running them for storage, ££££ these days).

I had assumed too that if I shutdown / lost another host then everything would come to a halt, but there does appear to be some info 'out there' about a 2 node set up and avoiding split brain etc. so I was hopeful it may be possible!

So really the only 2 node option is XOSAN which is kind of not an option by the sounds of it after Gluster going EOL and for playing at home needing to pay or manual install which I don't think there's a guide for (not moaning, you guys give a lot). I guess my aim is to have a 2 host set up, which I can reduce to 1 when I don't need to capacity or to allow the installation of updates without getting shouted at..... 'whys the internet not working' . At present I'm running Ceph hyperconverged in VMs (for VMs and Kubernetes storage........ I know......) across 3 hosts and in reality often run with 2. Yeah I know that if one goes down everything stops, but if I plan to, I just start the third, let it sync and sort itself out and everything is good to do whatever I want. Likewise if something does go wrong, starting the 3rd hosts often gets things moving again whilst I work it out. I really think that I should have lost some data by now by doing something silly, but so far (a few years now) it just sorts it all out. I also rsync everything to TrueNAS every hour and regularly backup the VMs just in case.

I guess I just need to accept that going to two hosts probably means that when moving VMs around their storage needs to go with them, and that TrueNAS (a VM again...... I know, but must be 15+years with no issues) needs to provide the storage for Kubernetes

Just got this working in my 3 host home setup..... But I'm looking to drop down to two hosts. Is it going to be usable with 2 hosts (I've seen the recommendation of 3+ at the top) and if so, what happens when you get down to 1 host whatever reason??? Are read / writes locked on the remaining host?

Been interested in this since the first blog post but never got round to trying it. Is this still a 'thing'? I'm having little luck trying to get it work and am seeing most of the errors already posted, and have tried the fixes but still no luck. I would guess I may have a template issue.......

Mainly wondering if its worth some effort or if its best to just run docker in a VM?

Thanks.

I get this too with some back ups but not all. Not had chance to work out what results in what though

It appears that the rollback worked, but doesn't appear to have an impact on my issue, which is good news in a way.

@stormi thanks as always. If I do find the issue I'll let you know..... if I break everything then I'll just get a

Hi @stormi is there any easy way to rollback the September updates??? I'm guessing not but I have a strange issue which I think coincides with my last reboot after applying the updates so want to confirm or eliminate as the cause.

Thanks.

Just rebuilt to xo-server 5.81.2 / xo-web 5.86.0

Removing the Proxy from the remote and it looks like everything is good. Restored backup taken in Dec 2019.... so plenty of versions ago, and all is ok.

Backups still work with the Proxy.