@olivierlambert
Merci Oliver (& Stormi), I found the issue was in front of the computer (again): I had missed to clone the repos before starting the run.py.

Seems to be working just now, at least it's compiling a ton of stuff....

@fred974

On the virtual machine advanced attributes tab you can chose between a RealTek RTL8139 and an Intel e1000 (both virtual devices) NIC. It defaults to RealTek and the suggestion is to change it to the Intel one and retry booting the VM.

The NIC on the host (X520) does not matter at all for this operation.

It's been awhile, but I can confirm that moving images between (in my case) oVirt and Xcp-ng VMs via Clonezilla works just fine.

Also did it between VMware and Xcp-ng btw.

@fred974
Nice to hear!

Yes, not having a template or using a wrong one seems to be an issue for Xcp-ng, when really it should be nothing but an easier way to set presets.

I guess one of these days I'll simply have to investigate what's in them and how difficult it would be to create your own.

Clonezilla has failed me on machines with the more complicated LVM and thin-allocation setups e.g. when trying to virtualize physical machines. But when it comes to cloning VMs between hypervisors it pays off that I tend to keep these simply and have the hypervisor deal with sparsity and overcommit.

Clonzilla certainly has helped me out a lot of times and I'm really glad they maintain it.

@fred974

On the virtual machine advanced attributes tab you can chose between a RealTek RTL8139 and an Intel e1000 (both virtual devices) NIC. It defaults to RealTek and the suggestion is to change it to the Intel one and retry booting the VM.

The NIC on the host (X520) does not matter at all for this operation.

It's been awhile, but I can confirm that moving images between (in my case) oVirt and Xcp-ng VMs via Clonezilla works just fine.

Also did it between VMware and Xcp-ng btw.

@Andrew

Thanks for your response!

In the mean-time I've finally found the hints on how to work around the USB NIC renaming issues, both on the forum (even directed at me, but somehow not read) and by Eric Eikrem on his site, so I'll try that next to make the r8156 2.5Gbit USB3 NICs work (got lots of those) on the Atom boxes.

I'm not touching the NUCs (for igb/e1000 testing) at the moment, because I need them very stable to play with LINSTOR without a VGA console.

Just to illustrate: for weeks my NUC10 would disappear from network after a couple of days without issues and even if it was still visibly running (normal HDD LED activity) nothing but a hard reset would bring it back online. Just couldn't understand what was going on and if it was some type of hardware issue with the box (just out of warranty).

In the end it was one of the myriad of BIOS settings, could have been 'modern standby' or ASPM, which was reactivated after a firmware update and caused these problems days later.

@Andrew
I'm messing around with all sorts of things:

• Loss of video output on Gen10/11 iGPUs and Ryzen 3 (Cezanne) iGPU during the Xen-Dom0 handover (may be a grub issue with the UEFI frame buffer driver). It's on hold, for lack of time and because Xen doesn't make hacking boot stuff any easier and it would take me weeks I do not have to get deep enough. Also, apart from the missing console, the machines work just fine, after I transplant the installed system from the NUC8 to the other targets.

• Lack of IOMMU support on my Ryzen 9 5950X with an Nvidia RTX 2080ti. It's been judged a BIOS issue here, but it works just fine with KVM and VMware. I've been going through the code, but unless I get kernel debugging going during the boot phase with a serial console, there is little chance of tracking down what's going on. From the logs alone, the code simply can't find the IOMMU device, or rather the data structures that describe it, but it's a ton of barely readable deeply cascaded spagetti of #define 'function calls'... written by an AMD guy, so XenSource will most likely point fingers, not invest into a work-around. Funny thing there is that this very system had been the first to run Xcp-ng in my lab, using a nested setup with VMware Workstation on Windows 2019 as a base....

• Support for RealTek r8156 USB3 2.5 Gbit Ethernet adapters: I use those on Pentium Silver J5005 based passive mini-ITX machines with oVirt and want to transition them to something still alive. I got a version of the driver that compiles and works just fine, but Xensource uses all kinds of tricks to rename NICs to be consistent across a pool that might have widely different (and in the case of USB NICs, dynamic) device names assigned to NICs. Currently the Citrix code at the base of interface-rename cannot deal with NICs that aren't connected (directly) to the PCI bus. It doesn't look for USB devices and thus the bridge creation and the overlay network stuff just fails to use the device. I guess the only sensible thing to do is to open a ticket at XenSource and see if Andrew Cooper, who seems to have written all the xcp Python bindings, will incorporate USB NICs ...which I doubt, given the giant amount of extra support trouble hotplugging NICs might bring about, when there is no revenue in this space. Would be an interesting test of XenSource collaboration dynamics to see if an xcp-ng based addition would be accepted upstream

If you're confident hacking XenSource Python2 library scripts, have a look at /lib/python2.7/site-packages/xcp/pci.py class PCIDevices (line 259), where it's using lspci -mn to find NICs.

I've also been looking a bit at support for the newer Intel NICs, which are built into my NUC10 and NUC11 devices, which aren't supported by the 4.19 kernel e1000e device driver. Again, it's not a priority for me, because I am using TB3 connected Aquantia 10GBase-T NICs for these faster NUCs with NVMe storage, as they are just the better match and literally zero trouble, if you disable the onboard NICs before installing.

The technical evolution in the mobile/desktop based edge appliance space currently is at a pace that completely overwhelms the XenSource roadmap, even Linux itself in many ways, because only Windows support sells that hardware. It's a bit of a nasty turn on NUCs, which for many years were a nicely conservative platform with great Linux support and plenty of efficient power for the home lab.

I have used an Nvidia GTX1080ti for pass-through testing, which also combines the GPU and a USB-C controller. It works just fine for CUDA stuff, (remote) gaming via Steam on Windows isn't great but proves pass-through is working.

The only issue I had was that I had to make sure that I was putting both devices in a single line for the boot flags, the Dom0 delete and the DomU add (xe vm-param-set), and not add them line after line, as evidently these statements are not cumulative.... I wound up with only the USB controller being visible inside the VM until I re-read the documentation.

I also had nothing connected the USB controller at the time, which might easily be an issue otherwise.

Did you check that the pass-through devices were truly gone from Dom0 before you tried adding them to the VM?

Then it's evidently 2 GPUs with a USB-C controller each: were all 4 devices passed through, or was it perhaps a mix across the boards?

@Kaiz

I am very much guessing here, but: I don't think you're looking at a single card in this case. Rather must be a switch of some kind involved, which translates between PCI and PCIe and also allows more than one PCI device to be connected to the PCIe bus.

And you'd have to

• remove all devices from the Dom0 (host)
• add all devices to the DomU (guest) VM (as a list in a single command)

lshw may help you figure out the bus topology and device IDs, although none of the Linux tools seem to come close to how HWinfo displays things on Windows. It should definitely help you see things more clearly on the Windows VM, once devices arrive there.

If you have any other device, say a spare NIC to test on first, you may be able to save yourself pulling from pulling your hair out.

@stormi

Thanks Stormi, that's true and it does seem to include the patched source...

So I wonder (just a bit): What's the difference between the two?

But I suspect it's simply some rpmbuild magic...

@olivierlambert

No matter if I am building xen, xapi or the kernels, there is one issue for which I'd like some help:

Just before the build finishes, all the sources used for the build get deleted, even when using the --no-exit option to stay in the container.

E.g. the final lines of the kernel build are like this:

Processing files: python2-perf-alt-4.19.227-1.xcpng8.2.x86_64
Executing(%license): /bin/sh -e /var/tmp/rpm-tmp.kWRZph
+ umask 022
+ cd /home/builder/rpmbuild/BUILD
+ cd kernel-4.19.19
+ LICENSEDIR=/home/builder/rpmbuild/BUILDROOT/kernel-alt-4.19.227-1.xcpng8.2.x86_64/usr/share/licenses/python2-perf-alt-4.19.227
+ export LICENSEDIR
+ /usr/bin/mkdir -p /home/builder/rpmbuild/BUILDROOT/kernel-alt-4.19.227-1.xcpng8.2.x86_64/usr/share/licenses/python2-perf-alt-4.19.227
+ cp -pr COPYING /home/builder/rpmbuild/BUILDROOT/kernel-alt-4.19.227-1.xcpng8.2.x86_64/usr/share/licenses/python2-perf-alt-4.19.227
+ exit 0
Provides: gitsha(ssh://git@code.citrite.net/XS/linux.pg.git) = cb3c28f7e8213ef44e5c06369b577a18b86af291 gitsha(ssh://git@code.citrite.net/XSU/linux-stable.git) = dffbba4348e9686d6bf42d54eb0f2cd1c4fb3520 python2-perf-alt python2-perf-alt = 4.19.227-1.xcpng8.2 python2-perf-alt(x86-64) = 4.19.227-1.xcpng8.2
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.4)(64bit) libc.so.6(GLIBC_2.4)(64bit) libc.so.6(GLIBC_2.7)(64bit) libc.so.6(GLIBC_2.8)(64bit) libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) libpython2.7.so.1.0()(64bit) python(abi) = 2.7 rtld(GNU_HASH)
Conflicts: python2-perf
Processing files: kernel-alt-debuginfo-4.19.227-1.xcpng8.2.x86_64
Provides: kernel-alt-debuginfo = 4.19.227-1.xcpng8.2 kernel-alt-debuginfo(x86-64) = 4.19.227-1.xcpng8.2
Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/builder/rpmbuild/BUILDROOT/kernel-alt-4.19.227-1.xcpng8.2.x86_64
Wrote: /home/builder/rpmbuild/SRPMS/kernel-alt-4.19.227-1.xcpng8.2.src.rpm
Wrote: /home/builder/rpmbuild/RPMS/x86_64/kernel-alt-4.19.227-1.xcpng8.2.x86_64.rpm
Wrote: /home/builder/rpmbuild/RPMS/x86_64/kernel-alt-headers-4.19.227-1.xcpng8.2.x86_64.rpm
Wrote: /home/builder/rpmbuild/RPMS/x86_64/kernel-alt-devel-4.19.227-1.xcpng8.2.x86_64.rpm
Wrote: /home/builder/rpmbuild/RPMS/x86_64/perf-alt-4.19.227-1.xcpng8.2.x86_64.rpm
Wrote: /home/builder/rpmbuild/RPMS/x86_64/python2-perf-alt-4.19.227-1.xcpng8.2.x86_64.rpm
Wrote: /home/builder/rpmbuild/RPMS/x86_64/kernel-alt-debuginfo-4.19.227-1.xcpng8.2.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.iAlDiX
+ umask 022
+ cd /home/builder/rpmbuild/BUILD
+ cd kernel-4.19.19
+ /usr/bin/rm -rf /home/builder/rpmbuild/BUILDROOT/kernel-alt-4.19.227-1.xcpng8.2.x86_64
+ exit 0
~

and my problem is with the %clean section, which removes the patched source code, which I'd really love to read, because it's not available as plain source in any repository, only as a mix of upstream source repos and Vates patch files.

I've been trying to find out how I can avoid the %clean section from being executed as part of rpmbuild, but I've failed to find whence this ultimate /usr/bin/rm -rf [...] is coming from or how to suppress it.

@TheFrisianClause

From my experiments using a GTX1080ti you'll have to follow the instructions for generic pcie-passthrough to the letter, which mostly means that the passthrough needs to be done on both sides, on the Dom0 for relinquishing device control and for the DomU to pick up the device (section 5). Perhaps now that the restrictions from Nvidia have gone, Vates will kindly include some GUI support for those operations in XOA.

Note, that if your dGPU has multiple devices (e.g. my GTX 1080ti also has a USB-C controller on-board), both entries need to be added in a single 'xe vm-param-set' statement, otherwise only the latter device (USB in my case) will wind up in the VM.... (yeah, at least 30 minutes of puzzling on that one)

Of course, if the dGPU is your console device, it means flying blind afterwards, but I'm getting used to that with all the recent iGPUs as well (then again, I have some DisplayLink hardware that's currently unused and EL7/8 drivers for those have popped up recently...)

Thankfully the dreaded error 43 issues have gone away with the more recent Nvidia drivers, sadly Kepler support has been retired (got a lot of those still around), so you may want to preserve the latest CUDA 11.4 release as one that offers both, but for Maxwell you should still be fine.

Before trying to diagnose with the Nvidia drivers, you should be able to see the device transition via lspci on both sides, Dom0 and DomU.

@olivierlambert
Merci Oliver (& Stormi), I found the issue was in front of the computer (again): I had missed to clone the repos before starting the run.py.

Seems to be working just now, at least it's compiling a ton of stuff....