99dea17e-ae5a-4e4e-a2a9-5f7eab98904f-image.png

@Greg_E

Also I should say that I've rebooted many times and still OK, going to leave it baking while I get on to other things.

@john-c @olivierlambert Now we're talking!!!

Here are the results of a 2 VM Delta/NBD backup (initial one) using 2 10GB NICS in bond:

ScreenShot 2024-11-18 at 13.22.56.png

WHAT a difference, when we run XOA on an Intel host instead of an EPYC one, with backups.

I've told this from the beginning, that the slow backup speeds were due to the EPYC issue (as I got 200+ MB/s @home with measily Protectli's on 10G)

Looking on what the Synology gets: I get up to 500+ MB/s during the backup!

@Teddy-Astie Yes...
xcp-oi-interfaces.jpg

@high-voltages Just wanted to thank you again, its working now with the commands i did in the screenshot.

@olivierlambert thank you for the quick answers, just to put it into context the plan was to use accelerated Wireguard VPN in pfSense+ which comes with own FreeBSD optimized kernel modules:

Some more info on CPU extensions / QAT / IPSsec-MB / AES-NI related to that:

https://github.com/intel/intel-ipsec-mb https://docs.netgate.com/pfsense/en/latest/hardware/cryptographic-accelerators.html https://networkbuilders.intel.com/docs/networkbuilders/intel-avx-512-and-intel-qat-accelerate-wireguard-processing-with-intel-xeon-d-2700-processor-technology-guide-1647024663.pdf https://2023.asiabsdcon.org/program/_p05b/paper.pdf https://forum.opnsense.org/index.php?topic=37808.0 https://forum.netgate.com/topic/173606/solved-intel-qat-driver-with-wireguard-support/9 https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/dev/qat/include/common/adf_accel_devices.h#L38

Example CPU's having the relevant features incl. AVX-512:

https://ark.intel.com/content/www/de/de/ark/products/226102/intel-xeon-d-1713nt-processor-10m-cache-up-to-3-50-ghz.html https://ark.intel.com/content/www/de/de/ark/products/226113/intel-xeon-d-1722ne-processor-10m-cache-up-to-2-70-ghz.html

Some older Atoms have Intel SSE4.2 but miss the AVX-512 which i think bring the most performance:

https://ark.intel.com/content/www/de/de/ark/products/97937/intel-atom-processor-c3558-8m-cache-up-to-2-20-ghz.html https://ark.intel.com/content/www/de/de/ark/products/134803/intel-atom-processor-c5115-9m-cache-2-8ghz.html

Best regards

You can redirect its serial port to a TCP port on the host (xe vm-param-add uuid=<uuid> param-name=platform hvm_serial=tcp::7001,server,nodelay,nowait) then connect a Windbg remote kernel debugger using a connection string (com:ipport=7001,port=192.168.1.xx)

Ah great news, thanks for keeping us posted. Indeed, there's a large Xen version bump between 8.2 & 8.3 🙂 Great job for the investigation and the patch!

For migration between vendors (AMD/Intel) I suggest the warm migration feature of XOA, solving the problem with only few minutes of interruption.

This is absolutely something to write about in the release note of 8.3.

Agreed, this should be in the release notes. I'll add it.

It's now in the release notes: https://docs.xcp-ng.org/releases/release-8-3/#pool-metadata-backup-temporarily-unavailable-from-console

Passing through the usb adapter I am getting roughly 30ms response times which is acceptable. Definitely not as fast as the m.2 version but good enough to keep me from buying a mini-pc or SBC. I would love to switch to the m.2 if someone could post here if they get a success. Just a strange side note, I couldn't get it to list in xoa and use the gui to do the passthrough. I had to manually hide it from the dom and pass the usb adapter to the vm. I couldn't figure out how to refresh the pci list and tried doing reboots to see if it would pick up the new device but no luck. If anyone knows how to refresh that list I'd be interested.

Thanks,
SFD

@633kD4d
Same issue here trying to pass through a UPS usb device to a Linux guest.

VM won't start with VUSB active and the same error about fail to reset.

i also performed a clean install of 8.2, same power on issue when a vGPU profile is attached.

@Ascar said in Neofetch on XCP-NG:

@Forza Not a bash script, a package that requires installation. XCP-NG has all repos wisely disabled. That's why installing it on this specific type of host is not a trivial task.

Sure it is packaged for many distros, but it is just a bash script.

Neofetch is a command-line system information tool written in bash 3.2+

https://github.com/dylanaraps/neofetch

Neofetch only requires BASH, the rest of the dependencies are entirely optional

https://github.com/dylanaraps/neofetch/wiki/Dependencies

But to state what others already said, it is not recommended to install stuff in dom0.

@olivierlambert Thank you!

Do you actually "feel" the difference in terms of VM performance for your services?

OF COURSE! Was very slow.
You cannot imagine the difference...

This is an AMD Ryzen 9 7950X3D host.
Moreover, since the problem started we migrated several VMs to other hosts.
If you see the screenshots above, in xentop output, besides Dom0, all other VMs where like hardly using 2-3 CPUs in total.
And everything was dead slow (steal time always high).

My feeling here is that this was somehow related to networking (tap*), but really don't know what and why.
Glad this was resolved after the reboot.

First of all, sorry for hijacking this old topic but as it was so well documented, it made more sense to expand upon it then to try and recreate everything that was already here.

Second, I generally don't use XCP-ng Center. However, once coming across this post I did open it to verify that what I saw in this post was the same as what my XCP-ng Center was showing.

Third, I am on one of the latest community additions but I have seen this occur a couple of time on previous versions as well.

Okay with that out of the way, I now believe that I know what is causing this to happen...

When you have a VM with multiple VDI's in it and one of those VDI's is both large and fairly busy, if the following exists then XO will not show one or more of the smaller drives despite the hosts still functioning normally while working on that VM:

The virtual machine has multiple snapshots of all of the VDI's in it. (IE like after several backups) For some reason a bunch of the snapshots are consolidated at one. All the snapshots that were consolidated are removed from the VM and the snapshots of the VDI's have coalesced but the system is still working on coalescing the large VDI. If a backup process tries to work on this VM, it will create a snapshot that XO shows to stay connected to the smaller drives even though the backup is skipped because a new snapshot of the large VDI can't be created due to the older snapshots not being coalesced yet.

Once the large VDI has properly coalesced all it's snapshots, the hosts will continue to treat it as any other VM but that VDI never does show up in XO again.

The only solution that I have found to fix it is to turn off the VM and then create a copy of missing VDI or VDI's and use those copies to replace the missing VDI's.

I know that I am just a member of the community but I hope that this helps everyone to understand what is going when this happens to them. 😉

It happened to me a couple of times before I was able to peace together the whole puzzle.

Topic restored. @gtuminauskas Please don't remove threads that have been responded to by other forum members.

@Affonso Looks like it might have been Bug 15684 in 24.03 that was resolved for 24.11 (release notes).

@dsmteam Still trying to browse the web and various xo forum but it looks like those parameters are in the .c and other precompile file so the build in xcp-ng are probably using those default parameters.