@olivierlambert said in Epyc VM to VM networking slow:

No obvious solution yet, it's likely due to an architecture problem on AMD, because of CCDs and how CPUs are made. So the solution (if there's any) will be likely a sum of various small improvements to make it bearable.

I'm going to Santa Clara to discuss that with AMD directly (among other things).

Do we have other data to back this? The issue is not really common outside of Xen. I do hope some solution comes out from the meeting with AMD.

@olivierlambert Nice Thanks for the feedback.

@DustinB Yes, I remember it being like this. However, it would be nice if it wasn't So I'd see this as a feature request.

When looking at the VM->Backup screen, the status (FAILED/OK) is for the Backup as a whole, and not specifically the VM itself, which is a little misleading.

In this screenshot we see Failed, but this specific VM was in fact backed up properly.

Using XOA Premium, Stable channel.

@CodeMercenary

I normally install uptades on the pool master, shutdown VMs, reboot. Then i do the same procedure on pool members.

@Chico008 easiest is to switch to another tab, like stats, and then back to console.

xcp-ng center might be an alternative as works better in this regard.

Using XOA "Disaster Recovery" backup method can be a lot faster than normal offline migration.

One time I did it, it took approx 10 minutes instead of 2 hours...

@DustinB said in Delta Backups:

@IanMcLachlan Are you trying to run multiple types of backups in the same job? Can you show us your "Backup Jobs" overview page?

I think he is referring to the full backup interval option: https://docs.xen-orchestra.com/incremental_backups#key-backup-interval

Enabling various custates and frequency scaling in BIOS/Firmware can help powr consumption. There is some latency cost as it takes longer time from deeper states.

Linux also has CPU frequency governors, but I am not sure how the Xen kernel handles this. Remember that dom0 is a VM under Xen, so things aren't as simple as with plain bare-metal Linuxes.

There's some information here about this:
https://wiki.xenproject.org/wiki/Xen_power_management

@TeddyAstie That is interesting. I had a look. The default seems to be cubic, but bbr is available using modprobe tcp_bbr. I also wonder if different queuing disciplines (tc qdisc) can help. For example mqprio that spreads packes across the available NIC HW queues?

@olivierlambert said in Migrating an offline VM disk between two local SRs is slow:

80/100MiB/s for one storage migration is already pretty decent. You might go faster by migrating more disks at once.

I'm not sure to understand what difference are you referring too? It's always has been in that ballpark, per disk.

This is not over a Network, only between local ext4 SRs on the same server. I tried the same migration using XCP-ng center and it is at the moment double as fast:

Can't really see any difference though. It is the same sparse_dd and nbd connection Perhaps it's a fragmentation issue. Though, doing a copy of the same VHD file gives close to 500MB/s.

@DustinB said in Migrating an offline VM disk between two local SRs is slow:

Separate question, why are you opting to use RAID 0, purely for the performance gain?

Yes, performance for bulk/temp data stuff.

@DustinB, no these are local disks not networked disks. I used to get around 400-500MB/s or so for plain migration between the two SRs.

@jebrown you could export xva or ova copies. But then those are not exactly identical to backups.

Hi!

I had to migrate a VM from one local SR (SSD) to another SR (4x HDD HW-RAID0 with cache) and it is very slow. It is not often I do this, but I think that in the past this migration was a lot faster.

When I look in iotop I can see roughly 80-100MiB/s.

I think that the issue is the sparse_dd connecting to localhost (10.12.9.2) over NBD/IP that makes it slow?

/usr/libexec/xapi/sparse_dd -machine -src /dev/sm/backend/55dd0f16-4caf-xxxxx2/46e447a5-xxxx -dest http://10.12.9.2/services/SM/nbd/a761cf8a-c3aa-6431-7fee-xxxx -size 64424509440 -good-ciphersuites ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-GCM-SHA384:AES256-SHA256:AES128-SHA256 -prezeroed

I know I have migrated disks before on this server with multiple hundreds of MB/s, so I am curious to what is the difference.

This is XOA stable channel on XCP-ng 8.2.

My wishlist for a new XCP-ng:

• Recent kernel for dom0.
• Guest trim support -> i.e. guest trim translates to shrunk VDI file.
• Better VM console support: Spice and/or RDP support with client USB access, shared clipboard and file transfers. Note, I do not mean remoting into the guest itself, but providing console access via XCP-ng like the current console VM access is with XO and XCP-ng Center.
• Implement VirtIO support: virtio-net, virtio-gpu, virtio-blk/scsi. (There are also serial and socket virtio devices, but i do not personally use these much, but could be important for management of some types if VMs).
• Support multiqueue i guests for net and blk.(would be possible with virtio)
• VirGL support. This is important as an alternative to GPU pass through or SR-IOV. It would in theory support migration too.

@Tristis-Oris We've had the same problem, so are not using CBT for now.

@mauzilla I do not think numa is exposed to the guests, so they will only see the number of cores assigned. I. E., you can migrate them just fine.

It's a good discovery that having XOA outside the pool can make the backup performance much better.

How is the problem solving going for the root cause? We too have quite poor network performance and would really like to see the end of this. Can we get a summary of the actions taken so far and what the prognosis is for a solution?

Did anyone try plain Xen on a new 6.x kernel to see if the networking is the same there?

Check that XO and XCP-ng uses a ntp server.