From LINBIT's description, drbd_peerdevice_outofsync_bytes is what one node's bitmap says is still out of sync with one particular peer, while the UpToDate in linstor r l is the state of each node's own disk, so as far as I understand the two aren't measuring the same thing (the resource states are described at https://docs.xcp-ng.org/xostor#drbdlinstor-resource-state).

Your table has the 119 GiB on .10 and .14, the two Unused copies, so I'd be curious which peer those series point at; the labels on the metric should say. If I'm reading drbd-utils right, drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics on xen01 also prints an out-of-sync figure per peer, but I haven't got an XOSTOR cluster to try it on, so treat that as a (not so educated) guess.

Whether a non-zero number between two secondaries means anything is beyond me; @Team-Storage will know.

From what I read in the docs (HA isn't my strong suit), a host in an HA pool that loses its heartbeat in certain ways is designed to reboot itself, which they call self-fencing, so this may well be HA doing its job rather than something crashing.

The log you attached is from the master and starts at 11:21:00, and dacshyp001 is already marked as not live in the very first liveset at 11:21:08, so I think whatever triggered it happened just before that and isn't in this file.

The three Setting host dacshyp001 to dead lines look to me like one event being re-checked every 20 seconds while 001 was still coming back, though I could be misreading that.

I also suspect 003 went down at a different moment, because it still shows as alive in that same liveset. There's a doc section for this case, https://docs.xcp-ng.org/troubleshooting/troubleshooting-ha#my-host-rebooted-why-did-it-reboot, which points at /var/log/xha.log on the host that rebooted.

Could you post that file from dacshyp001 and dacshyp003 for the few minutes before each reboot, plus 003's boot time?

6.8.2 went out, and it carries florent's proxy patch, so the thing that was stopping you switching channels has cleared.

Before you do, though, the 6.8.2 changelog lists exactly one bug fix, the fetch failed proxy one, with xo-server at 5.208.3 and the proxy at 0.31.10, and there's nothing in it about metadata backups or BodyTimeoutError.

Your own post above points the same way, since you're seeing the stall on sources at the latest commit, and that commit already has the proxy fix in it. So I'd treat the metadata timeout as a separate bug that 6.8.2 won't touch, and I couldn't find an open issue covering it, though I might well have searched the wrong words.

Might be worth a mention to @Team-XO-Backend, mostly so the metadata half doesn't quietly get filed as fixed by 6.8.2.

You already spotted this over on 12468, so this is just to close the loop here.

That one's fixed: Pierre Brunet put the fix on a branch, @Andrew and @abudef both tested it, and it's merged now. Since you're running from source on master, you should be able to pull past that merge rather than wait for a release.

One thing to keep in mind, though: the same error string showed up in 11875 for a completely different cause, so if it comes back after you update, it's probably not this bug again. I'd let 12468 be the thread to follow since that's where the people who fixed it are.

You're not blind, and your fingerprint is right. I pulled packer-plugin-xenserver_v0.11.4_SHA256SUMS.sig off the GitHub release and ran gpg --list-packets on it, and it reports issuer fpr v4 0DD2E92A7A0F682BC29F87073084D82948625D89, which is exactly what you pasted.

It doesn't match the XCP-ng key because it isn't ours.

The plugin our docs point at is ddelnano/packer-plugin-xenserver, maintained outside Vates, so it's signed with the maintainer's key while 34AC2EB6... only signs XCP-ng RPMs.

I couldn't find that key published anywhere either. It's not on keys.openpgp.org or keyserver.ubuntu.com, it isn't in the repo, and it isn't among the 11 keys at github.com/ddelnano.gpg.

That looks like something worth raising on the plugin's own issue tracker, and you'd file it better than I would since you're the one trying to verify a download. I'd also say our IaC docs could stand to mention the plugin isn't ours, which is what sent you looking at the wrong key in the first place.

@abudef asked almost exactly this a few days ago over on 12466, and @pierrebrunet answered it there with a link to the doc describing the layout, so that thread's probably your quickest route.

For the specific "which VM is this folder" question, the UUID in the directory name is the VM's own UUID, so xe vm-list uuid=<that-uuid> params=name-label on a host should give you the name back.

One thing that bite me when I tried it: if the VM has since been deleted, that command prints nothing at all and still exits cleanly, so silence means "no such VM here" rather than "the command didn't work".
Those silent ones are probably where your space has gone, if I had to guess. I'm not the backup expert here so someone may well have a tidier way of doing it.

Thanks for doing that, @john.c.

The sponsorship side isn't mine to speak to; I forwarded to the right person

The CFP I can say something about: nor(DEV):con is already in the conference pipeline I keep an eye on, and their Sessionize call closes on 30 September, so your timing is better than you probably realised. I can't promise you'll see us in Norwich, but the nudge landed in front of the right people.

Forwarded to the right team, thanks!

Thanks a lot for your feeback, folks!

I gave this a go on my lab XOA, and XO isn't touching your option at all: the failure quotes the command it ran, mount -o sec=krb5p -t nfs <server>:/path /run/xo-server/mounts/<remote-id>, and a control remote with no custom options mounted the same export fine.

It does fail with mount.nfs: an incorrect mount option was specified, but that line tells you less than it looks like: I fed it sec=totalnonsense and got the identical message back, which is why it reads like XO refusing a valid option when it's really just relaying what mount.nfs said.

The appliance ships rpc.gssd as part of nfs-common, so that part's there. What isn't there is /etc/krb5.keytab or /etc/krb5.conf, and the systemd unit carries ConditionPathExists=/etc/krb5.keytab, so the daemon never starts. Mine last failed that condition at boot eleven days ago and said nothing about it.

I went one step further: dropping a keytab in place is enough for rpc.gssd to start and stay up, so that condition really is the only thing stopping it, and it doesn't need krb5.conf for that. Whether the mount then works needs a KDC and principals that agree with each other, and I couldn't get that far, so that part is still untested.

Thanks for pasting the whole log; that made this much easier to chase.

Wild guess, but I think this may be a regression from PR #10124 ("replace vhd-lib calls with abstract methods"), which was squash-merged onto master as adecd7306 on 2026-09-11 at 09:42 UTC, just before your 202609112300 nightly was cut. master is still sitting at 99312eca2, with no fix (if any fix is needed, I'm out of my depth in this area) after that commit that I can find.

Reading the code on master, the transfer step now checks the VHD it just wrote and passes that data-directory path along while the .alias.vhd doesn't exist yet; RemoteVhdDisk.init then refuses a directory path that isn't an alias. I could easily be wrong about that, though, and thread 11875 is a reminder that this same message has had more than one cause.

If, by any chance, I've got it right, it would only affect repositories with "store backup as multiple data blocks" enabled. So does it also fail on a repository without that option?

Rolling back to a commit from before yesterday morning should get you going again in the meantime, and it's probably worth mentioning @Team-XO-Backend so the people who own this code see it.

And if I've made a total fool of myself with this "analysis", so be it.

Pretty cool, @samuelolavo, thanks for clarifying!

Nice work, and thank you for putting it out under MIT.

What happens to a client like this when the REST API moves under it? You have tagged 1.0.0 against 6.8.0 and 0.39.0, and I am curious whether you are planning to track versions or to fail loudly when something it expects has gone.

Thanks for the feedback, @dvinni !

Thanks a lot for the feedback, @christopher-petzel !

We are fully aware of Broadcom's decision to stop providing the VDDK library to the cloud community or to reserve its download to some authorized partners and customers. As much as we regret this decision, we have to respect it.

Because it impacts users who need to make backups, we are evaluating alternative options to address this situation. We will provide further guidance as this evaluation progresses but can't share more information for now.

On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising tcp_mem buys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on an xo_server restart points the same way.

It might be worth a mention to @Team-XO-Backend, since they would know whether xo-server keeps sockets per pool.

@johnnezero I'm relieved you took it that well.

Second plugin in a couple of months, you're tireless.

I don't want to dampen your enthusiasm, but... there's something you'll probably want to know: XO's load-balancer already covers part of this, affinity tags were merged upstream back in November 2025, and there's an open pull request adding VM-to-host affinity right now, which looks (to me, at least) like your AM-RunOn and AM-NotOn: https://github.com/vatesfr/xen-orchestra/pull/10207.

What's already there is written up at https://docs.xen-orchestra.com/xo5/load_balancing#vm-affinity, and XCP-ng has its own anti-affinity at https://docs.xcp-ng.org/management/vm-load-balancing#vm-anti-affinity.

I'm not saying that makes yours redundant, and I might be misreading how much of it really overlaps, but AM-KeepApart and the dry-run default look like they go past what's in either of those.

On one machine in my lab, /etc/rsyslog.d/xenserver.conf defines 17 outchannels, every one of them at 104857600 bytes, and /var/log/messages isn't one of them; it's written by the stock *.info;mail.none;authpriv.none;cron.none line in /etc/rsyslog.conf and rotated by /etc/logrotate.d/syslog, so nightly and with no size trigger.

The docs at https://docs.xcp-ng.org/guides/logs#rsyslog describe the 100 MiB rule but don't list which files it actually covers, which is probably why this is hard to work out without reading the config the way you did.

Whether messages can realistically outrun a nightly rotation I don't know, and I'd be guessing if I said either way; the one thing in your favour is that /var/log is its own filesystem, 3.9 GB on the box I looked at (small lab machine, I haven't searched in the code, so I don't know where that size is decided), so it can't take the rest of dom0 with it.

Whether that omission is deliberate or just inherited (or even something fancier) is really a question for whoever owns that file, so it might be worth a mention to @Team-OS-Platform-Release.