@Biggen At the moment, xcp-ng center provides some better views and overviews not yet available in XO.. Hoping next major version fixes this

@florent said in Long backup times via NFS to Data Domain from Xen Orchestra:

@MajorP93 this settings exists (not in the ui )

you can create a configuration file named /etc/xo-server/config.diskConcurrency.toml if you use a xoa

containing

[backups]
diskPerVmConcurrency = 2

That is great. Can we get it as a UI option too?

@olivierlambert said in XCP-ng Guest Agent - Reported Windows Version for Servers:

It's funny to see Microsoft having a version 10 for an edition named 11. I suppose it's not a surprise for an organization that huge.

They did say that Windows 10 would be the last version of Windows...

@dinhngtu Thank you. I think it is clear for me now.

The docs at https://xcp-ng.org/docs/guests.html#windows could be improved to cover all three options but also to be a little more concise to make it easier to read.

@iams3le we have switched to the signed xcp-ng drivers. We also replaced our older 2022 servers.

Tested the new updates on my prod EPYC 7402P pool with iperf3. Seems like quite a good uplift

Ubuntu 24.04 VM (6 cores) -> bare metal server (6 cores) over a 2x25Gbit LACP link.

Pre-patch

• iperf3 -P1 : 9.72Gbit/s
• iperf3 -P6 : 14.6GBis/s

Post Patch

• iperf3 -P1 : 11.3GBit/s
• iperf3 -P6 : 24.2GBit/s

Ubuntu 24.04 VM (6 cores) -> Ubuntu 24.04 VM (6 cores) on the same host

Pre Patch

Forgot to test this...

Post Patch

• iperf3 -P1 : 13.7GBit/s
• iperf3 -P6 : 30.8GBit/s
• iperf3 -P24 : 40.4GBit/s

Our servers have Last-Level Cache (LLC) as NUMA Node enabled as most our VMs do not have huge amount of vCPUs assigned. This means for the EPYC 7402P (24c/48t) we have 8 NUMA nodes. We however do not use xl cpupool-numa-split.

Sorry for spamming the thread.

I have two identical servers (srv01 and srv02) with AMD EPYC 7402P 24 Core CPUs. On srv02 I enabled the LLC as NUMA Node.

I've done some quick benchmarks with Sysbench on Ubuntu 20.10 with 12 assigned cores. Command line: sysbench cpu run --threads=12

It would seem that in this test the NUMA option is much faster, 194187 events vs 103769 events. Perhaps I am misunderstanding how sysbench works?

With 7-zip the gain is much less, but still meaningful. A little slower in single-threaded performance but quite a bit faster in multi-threaded mode.

Problem was a stale connection with the NFS server. A reboot of the NFS server fixed the issue.

@xcprocks said in Restoring a downed host ISNT easy:

So, we had a host go down (OS drive failure). No big deal right? According to instructions, just reinstall XCP on a new drive, jump over into XOA and do a metadata restore.

Well, not quite.

First during installation, you really really must not select any of the disks to create an SR as you could potentially wipe out an SR.

Second, you have to do the sr-probe and sr-introduce and pbd-create and pbd-plug to get the SRs back.

Third, you then have to use XOA to restore the metadata which according to the directions is pretty simple looking. According to: https://xen-orchestra.com/docs/metadata_backup.html#performing-a-restore

"To restore one, simply click the blue restore arrow, choose a backup date to restore, and click OK:"

But this isn't quite true. When we did it, the restore threw an error:

"message": "no such object d7b6f090-cd68-9dec-2e00-803fc90c3593",
"name": "XoError",

Panic mode sets in... It can't find the metadata? We try an earlier backup. Same error. We check the backup NFS share--no its there alright.

After a couple of hours scouring the internet and not finding anything, it dawns on us... The object XOA is looking for is the OLD server not a backup directory. It is looking for the server that died and no longer exists. The problem is, when you install the new server, it gets a new ID. But the restore program is looking for the ID of the dead server.

But how do you tell XOA, to copy the metadata over to the new server? It assumes that you want to restore it over an existing server. It does not provide a drop down list to pick where to deploy it.

In an act of desperation, we copied the backup directory to a new location and named it with the ID number of the newly recreated server. Now XOA could restore the metadata and we were able to recover the VMs in the SRs without issue.

This long story is really just a way to highlight the need for better host backup in three ways:

A) The first idea would be to create better instructions. It ain't nowhere as easy as the documentation says it is and it's easy to mess up the first step so bad that you can wipe out the contents of an SR. The documentation should spell this out.

B) The second idea is to add to the metadata backup something that reads the states of SR to PBD mappings and provides/saves a script to restore them. This would ease a lot of the difficulty in the actual restoring of a failed OS after a new OS can be installed.

C) The third idea is provide a dropdown during the restoration of the metadata that allows the user to target a particular machine for the restore operation instead of blindly assuming you want to restore it over a machine that is dead and gone.

I hope this helps out the next person trying to bring a host back from the dead, and I hope it also helps make XOA a better product.

Thanks for a good description of the restore process.

I was wary of the metadata-backup option. It sounds simple and good to have, but as you said it is in no way a comprehensive restore of a pool.

I'd like to add my own oppinion here. A full pool restore, including network, re-attaching SRs and everything else that is needed to quickly get back up and running. Also a restore pool backup should be available on the boot media. It could look for a NFS/CIFS mount or a USB disk with the backup files on. This would avoid things like issues with bonded networks not working.

Might a different solution be to use a USB network bridge instead of direct attached USB? Something like this https://www.seh-technology.com/products/usb-deviceserver/utnserver-pro.html (There are different options available)... We use my-utn-50a with hardware USB keys and it has shown to be very reliable over the years.

@olivierlambert said:

Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward.

My earlier ratios were too low: the rig was the bottleneck

Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all.

Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved.

arm	build	MiB/s	range	vs stock	stalls	B/seg
A stock	control	71.3	71.2-71.4	1.00x	78.3%	1441.6
B TCP_NODELAY only	+socket	303.8	303.1-305.0	4.26x	1.5%	1426.6
C NODELAY + pipelining	both	526.6	522.8-528.5	7.39x	1.1%	1425.7
D pipelining only	+pipeline	528.3	523.0-531.9	7.41x	16.1%	1442.0

Per-arm spread is 0.3% to 1.7%.

The full stack is worth 7.4x, not 5.51x, and TCP_NODELAY alone is worth 4.26x, not 2.80x.

The two changes are substitutes, not complements:

• TCP_NODELAY alone: 4.26x
• pipelining alone: 7.41x
• both together: 7.39x

So:

• Pipelining alone captures the whole win.
• Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing.
• The reverse is not true: pipelining on top of NODELAY is still worth 1.73x.
• NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput.

What the socket option still does after pipelining is remove the stall, 16.1% of samples down to 1.1%, for 1.14% more segments. On this rig that buys no throughput. See the RTT caveat before writing it off.

Correction 1: I said Chunked might regress. It does.

xe vdi-copy between two host-local SRs, 100 GiB, interleaved A/B/A/B. @TeddyAstie was right.

arm	transfer	data_segs_out	B/seg	stalls
A control	442.1 s	74,746,948	1438.5	0.0%
A2 control	439.4 s	74,677,058	1439.9	0.0%
B NODELAY	449.5 s	75,205,400	1430.1	0.0%
B2 NODELAY	455.5 s	75,260,858	1428.7	0.0%

Both controls beat both patched runs with no overlap, so this is not drift. TCP_NODELAY costs +2.7% wall clock and +0.70% segments here, and buys nothing.

Nothing, because the stall it fixes does not occur on this path:

• Sub-MSS stalls are 0.0% in every arm, including the unpatched control.
• Chunked writes a 12 byte header then data, with no per-request reply.
• So the send queue stays backed up (median notsent about 3.3 MB, rwnd_limited 99.7%).
• Nagle never faces a small-segment decision. Packing is already 0.993x MSS before the patch.

Where the extra packets come from, and it is not mainly the headers:

• The disk is fully allocated, so expand_copy (f.ml:2745) splits at its 2 MiB cap: 51,200 chunks.
• That is only 9.8% of the 521,126 extra segments.
• The rest is TLS record boundaries. The channel is unbuffered (channels.ml:129), so each 16 KiB TLS record is its own 16,413 byte write, 11.33x MSS, leaving a sub-MSS remainder.
• With Nagle those coalesce with the next record. With NODELAY they go out uncoalesced whenever the send queue drains.
• strace confirms both shapes: 41 byte writes are the headers (12 plus 29 of AES-GCM), 16,413 byte writes are the bulk records.

Data is correct either way: the patched copy md5s identical to the source, ce647d9436b48401cd4b489c955ef0f7.

Protocol confirmed two ways, for anyone reproducing:

• An https destination not advertising transfer-encoding: nbd yields [Chunked; NoProtocol] (impl.ml:1084), and the code takes List.hd.
• At runtime the receiver runs vhd-tool serve --source-format raw --source-protocol chunked --source-fd 8.

Correction 2: my reasoning for "NODELAY is a no-op after pipelining" was wrong

I said that with 8 requests in flight there is almost always an MSS queued, so the sub-MSS condition rarely arises. Not what happens. Arm D is pipelining with the socket patch removed, and it still stalls on 16.1% of samples against 1.1% with it. Depth 8 does not keep the queue above MSS.

The conclusion survives, the reasoning does not: NODELAY after pipelining gives no throughput gain because the stall stops being the limiting factor, not because the stall goes away.

Correction 3: the export path needs nothing

• Export runs stream_raw (impl.ml:335) : export_raw_vdi.ml:56 passes "none" as the destination protocol. No headers, no replies, so no write-write-read pattern.
• The patch could not cover it anyway: --destination-fd becomes fd://N, then File_descr, then Channels.of_raw_fd (impl.ml:991), never touching the socket helper.
• It does not need to. http_svr.ml:624 already sets TCP_NODELAY on the listening socket and Linux passes it to accepted sockets. Verified on the 4.19 kernel these hosts run, with a listener without the option as a control returning 0.
• stunnel sets it independently too (xapi_stunnel_server.ml:66-68).

The cost of TCP_NODELAY, every test we ran

path	Nagle on	TCP_NODELAY	cost
NBD 107 GiB, NVMe	1441.7 (0.996x MSS)	1425.2 (0.984x)	+1.16% segments
NBD pipelined, RAM	1442.0 (0.996x)	1425.7 (0.984x)	+1.14% segments
NBD 6-pair repeat, RAM	1441.9 (0.996x)	1424.6 (0.984x)	+1.20% segments
Chunked 100 GiB	1438.5-1439.9 (0.993x)	1428.7-1430.1 (0.988x)	+0.70% segments

About 1.2% more segments, reproducible to three digits across four independent experiments. Small, and it is the real permanent cost of the option.

Where the ceiling is now

At 7.4x we are at about 525 MiB/s, and it is not the obvious things:

• Not the network: 10G link, roughly 46% utilised.
• Not the sender: sparse_dd at mean 57%, peak 64% of one core.
• On the receiving host, tapdisk is roughly 72% of all busy CPU (python3 9%, xapi 6%, stunnel 6%).

That sample spans more than one leg and sums several tapdisk processes, so read it as dominance rather than a precise figure. It is where we would look next.

Our reading

Both changes are worth having, they are independent, and they are not additive, so the order is a real decision.

TCP_NODELAY

• Strongest argument is precedent, not the benchmark: vhd-tool is the only NBD client in the toolstack that leaves Nagle on.
• QEMU sets it unconditionally (nbd/client-connection.c:143) and forwards it through TLS. blktap treats failing to set it as fatal (drivers/block-nbd.c:793). xapi and stunnel both set it.
• 12 lines, 4.26x on code that ships today.
• Costs, now measured rather than assumed: about 1.2% more segments everywhere, and 2.7% wall clock on vdi-copy between two host-local SRs where it gains nothing.

Pipelining

• 7.41x on its own, and it makes the socket option irrelevant to throughput.
• Less work than it looks: the multiplexer already exists and is unused (nbd/lib/client.ml:78), so this is not the NBD redesign it first appears to be.
• The real obstacle is buffer ownership. expand_copy hands out slices of one shared 2 MiB buffer, so a naive pipeline silently corrupts data while reporting success.
• Our prototype works around it with a local buffer pool and a memcpy per block. The proper fix belongs inside expand_copy in f.ml, a shared library with other consumers.
• The prototype is not mergeable as it stands: progress counts issued rather than completed work, and a failed write leaves its siblings unawaited.

We will follow whichever route the XAPI team prefers and are happy to do the work either way. Our own order would be the socket option first, then pipelining done properly rather than our prototype. That is a sequencing preference, not a claim that the gains compose. If you would rather go straight to pipelining and skip the socket change, our data supports that: it reaches 7.41x on its own.

One caveat that cuts in favour of the socket option, and our rig cannot measure it:

• Every number here is from a 0.2 ms RTT link.
• The Nagle stall costs a delayed-ACK round trip, so its cost scales with RTT.
• 0.2 ms is therefore the regime least favourable to fixing it.
• On a higher-RTT link (cross-rack, cross-site, DR replication) the same 16.1% stall rate that costs nothing here should cost real throughput.
• So read 4.26x as a floor for TCP_NODELAY, and "pipelining makes it redundant" as a statement about short links specifically.
• We have not measured a long-RTT link and would like to hear from anyone who has.

Open questions

1. Is the 2.7% on vdi-copy acceptable, or should the option be scoped to NBD? Unconditional is simpler and matches every other component. Scoping avoids a measured regression on a path that cannot benefit.
2. Is the Chunked header worth batching regardless? It is a separate unbuffered 12 byte write before every chunk, which under TLS becomes its own 41 byte record. Writing header and payload together removes that independently of any socket option, and helps the Nagle-on case too.
3. Should expand_copy own a buffer pool? Any pipelining implementation needs it. Fixing it in f.ml fixes it for every consumer, but that is a wider blast radius than we wanted to take unilaterally.
4. Is the receiving tapdisk the next real ceiling? At 7.4x the sender and the network both still have headroom and tapdisk dominates destination CPU. We have not dug into why.
5. Does the picture change on a high-RTT or faster link? Both conclusions, that NODELAY is worth 4.26x and that pipelining makes it redundant, are properties of a short fast link that never saturated.

Rig

• Two host XCP-ng 8.3 pool, xapi 26.1.16, 10G, 0.2 ms RTT.
• Arms differ by exactly one setsockopt where relevant, same tree and toolchain.
• Every run verified by binary sha256 before it started, transfers checksum verified.
• RAM SRs are ext4 on a brd ramdisk. tmpfs does not work: no O_DIRECT, so VDIs can be written onto such an SR but never migrated back off.

Excellent work.

IMHO, disabling Nagle's algorithm is the best and least risky option. Application code paths and protocol behaviour stays exactly the same, except for the socket option. For a very risk free change we gain a lot or performance in a known bottleneck.

Pipelining exposes a new code path which needs more debugging and verification before it should be deployed, especially in a LTS situation.

Speaking as a paying enterprise customer, stability and predictably is important to us. Therefore I propose that the short-term solution is to disable Nagle's algorithm and ship it in the current LTS release as a normal bug fix.

Then, in a longer term, develop the Pipelining option and ensure it get proper testing and verification before release, particularly if backporting it to current LTS.

This is what I did.

• Migrated VMs off one host
• Disconnected that host from the pool
• Made a clean install of version 8.3 on that host and made into a new pool.
• Live-migrated grated VMs back to the new pool
• Made clean installs on the remaining hosts
• Joined the remaining hosts to the new pool

I re-checked again but the issue is unfortunately not resolved. It does not happen on all VMs and files, so maybe there is something wrong somehow in the VDI?

@McHenry

Looks like you want disaster recovery option. It creates a ready-to-use VM on a separate XCP-ng server. If your main server fails you can start the vm directly off the second server.

In any case, backups can be restored with XO to any server and storage available in XCP-ng.

@rtjdamen you could simply make two backup jobs, one for daily backups and one for monthly backups.

Since the nfs shares can be mounted on other hosts, I'd guess a fsid/clientid mismatch.

In the share, always specify fsid export option. If you do not use it, the nfs server tries to determine a suitable id from the underlying mounts. It may not always be reliable, for example after an upgrade or other changes. Now, if you combine this with a client that uses hard mount option and the fsid changes, it will not be possible to recover the mount as the client keeps asking for the old id.

Nfs3 uses rpcbind and nfs4 doesn't, though this shouldn't matter if your nfs server supports both protocols. With nfs4 you should not export the same directory twice. That is do not export the root directory /mnt/datavol if /mnt/datavol/dir1 and /mnt/datavol/dir2 are exported.

So to fix this, you can adjust your exports (fsid, nesting) and the nfs mount option (to soft) , reboot the nfs server and client and see if it works.

@MajorP93 aha, yea. Per disk concurrency is important too.

Hi,

I would like to have more progress information for the "Mirror backup" feature. Currently there is no progress other than how many VMs that have been copied so far.

What I would like is:

The total data size to be transferred plus how many VMs/backups that are left in the queue plus an approximate ETA. All this information should be available for XOA since it already knows what already exists in the source Remote.

Additionally I would like the option to cancel a running Mirror backup. The Cancel button is disabled when I try:

There is also no Task for the Mirror job listed the XOA Tasks screen.

I found an earlier question like this, but there was no answer there.
https://xcp-ng.org/forum/topic/11108/incremental-mirror-backup-progress-information-in-xoa

@Kennet You need to boot in UEFI mode. And secondly, you must make sure your nvme is using 512e insted of 4kn sector size. You can use nvme-cli command to check sector size.

@andrewm4894 said in Netdata package is now available in XCP-ng:

Qq, what would be the best way for me to try spin up a sort of test or dev XCP-ng env for me to try things out on? Or is there sort of hardware involved such that this might not be so easy. In my mind I'm imagining spinning up a VM lol which probably shows my level of naivety

You can run XCP-ng inside a VM, as long as the hypervisor underneath exposes nested virtualisation. The actual installation of XCP-ng is very easy. Mostly click and run.