I noticed you merged https://github.com/vatesfr/xen-orchestra/pull/9787
I just tried it. And it does seem to fix my original issue!
Thank you! I am always impressed by you guys. Making testing and reporting upstream (to you guys) a good experience!
J
Offline
-
RE: Building from source, now introduces local changes in typed-router.d.ts?
-
RE: Backblaze as Remote error Unsupported header 'x-amz-checksum-mode' received for this API call.
I submitted this as a GitHub issue last week.
TL;DR: Backblaze aparently doesn't support those flags that are enabled by default
"Backblaze does not yet accept these headers, so we recommend downgrading to AWS Javascript 3.x SDK version 3.728.0." -
RE: [Solved] SR_SOURCE_SPACE_INSUFFICIENT - Problems enabling HA
Thanks again for your input and recomendations! I'll verify that this is solved by having the LUN expanded to 8GB instead. Afterwards I'll mark your answer as the solution!
-
RE: Rolling Snapshot not cleaning up old snapshots, regardless of retention is set to.
@olivierlambert
I've finally had time to sit down and do some more troubleshooting.
It seems that the issue is somehow tied to the backup-jobs themselves.
Deploying XOA and subsequently importing XO-config from my XOCE instance. I continued to see several issues between both instances.
These issues pretty much went away, when I re-did the backup-jobs from scratch. And are now much more in-line with what I'm excepting to see. This was done on both XOA Stable, Latest and XOCEI am no longer able to repoduce this at all.
So my guess is that a bug of some sort got introduced with this being a constantly updated from source instance.Marking as solved. As I don't believe there is much more to do at this time.
-
RE: VM vCPU allocation
@McHenry
Differences could be things such as:- Amount of cores between hosts for migrations
- Guarantee performance between guests. So that they don't affect each other as much.
Same with overprovisioning dynamic RAM
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising
tcp_membuys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on anxo_serverrestart points the same way.
It might be worth a mention to @Team-XO-Backend, since they would know whether
xo-serverkeeps sockets per pool.This is why I'm not increasing the numbers quite yet. Since this would only mask a potential issue. If an underlying cause is responsible, then this would be of interest.
Thanks for chiming in!
-
RE: (Windows) guest IPv6 address doesn't collapse zeroes -> Long IPv6 addresses
I have done some more testing.
Windows Server 2022 is also affected. 9.1.145-77I also tried the latest WindowsPV drivers from Vates, on Windows Server 2025. 9.1.200-0. And the "problem" still persists.
It is a very low impact and low importance "problem". Since it is afaik purely costmetical.
-
RE: 🛰️ XO 6: dedicated thread for all your feedback!
I really miss the description column XO5 has in the default/home view. I understand space is limited for the tree view, but maybe it could appear when a host is selected in the tree view?
In XO6, you will see it if you click on the eye at the end of the row, then on the right panel.
Is this OK ?Personally I don't really find the rightmost pane/section all that usefull. It's to narrow to have any clear information. And it also leads to the other panels being smaller than they (imho) need to be able to show their base information as well.
This is especially noticable when using windows sizes narrower than 1080p.The rightmost pane I hope will get either removed, and redone in UI v7. Because right now XO5 is far more informational than that is.
Sorry for jumping on with my two cents.
-
RE: Building from source, now introduces local changes in typed-router.d.ts?
@MathieuRA
The sdn-plugin is back. I am amazed by that quick resonse and fix. THANKS!However, the issue with @xen-orchestra/web/typed-router.d.ts being locally modified during compilation, is still present. Leading to have to
git restore @xen-orchestra/web/typed-router.d.tsbefore being able to pull from github.The build-script that I use is based on (then) official docs
#!/bin/bash set -ex set -o pipefail git checkout . git pull --ff-only yarn yarn build -
RE: 🛰️ XO 6: dedicated thread for all your feedback!
Small discrepancy: VDIS or VDIs? Which one is it?
Red points out where it's called VDIS.
Green where it is called VDIs.And as a bonus feedback/question. Why is it called "A VDIS" ?(marked with yellow)
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising
tcp_membuys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on anxo_serverrestart points the same way.It might be worth a mention to @Team-XO-Backend, since they would know whether
xo-serverkeeps sockets per pool.This is why I'm not increasing the numbers quite yet. Since this would only mask a potential issue. If an underlying cause is responsible, then this would be of interest.
Thanks for chiming in!
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
Did a much closer look and can indeed confirm that the network capabilities go up from kb/s -> MB/s immediately from xo_server restarts.
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
@Forza
Comparing against the weaker XOCEStrong:
net.ipv4.tcp_mem = 90147 120196 180294 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304Weak:
net.ipv4.tcp_mem = 41766 55688 83532 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 -
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
Adding the initial memstats I was looking at whilst trying to troubleshoot
$ cat /proc/net/sockstat sockets: used 3478 TCP: inuse 37 orphan 0 tw 25 alloc 3149 mem 194631 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294Output of
sudo ssNetid State Recv-Q Send-Q Local Address:Port Peer Address:Port u_str ESTAB 0 0 * 10292 * 10293 u_str ESTAB 0 0 /var/lib/sss/pipes/nss 3260817 * 3261568 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12062 * 12838 u_str ESTAB 0 0 * 11638 * 11639 u_str ESTAB 0 0 * 13510 * 11079 u_str ESTAB 0 0 * 11624 * 11625 u_dgr ESTAB 0 768 * 10128 * 10127 u_dgr ESTAB 0 0 * 6593 * 6594 u_str ESTAB 0 0 * 5011 * 7454 u_str ESTAB 0 0 * 11692 * 11693 u_str ESTAB 0 0 * 11641 * 11642 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11295 * 11294 u_dgr ESTAB 0 0 * 6309 * 5237 u_str ESTAB 0 0 * 3261580 * 3260236 u_str ESTAB 0 0 * 3259924 * 3259925 u_dgr ESTAB 0 0 * 1806084 * 1806083 u_str ESTAB 0 0 * 12904 * 11125 u_dgr ESTAB 0 0 * 3545 * 3544 u_str ESTAB 0 0 * 3259925 * 3259924 u_str ESTAB 0 0 * 11666 * 11665 u_str ESTAB 0 0 /run/systemd/journal/stdout 7622 * 6577 u_str ESTAB 0 0 * 11660 * 11659 u_str ESTAB 0 0 /run/systemd/journal/stdout 7991 * 9524 u_str ESTAB 0 0 * 1806062 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 7914 * 7035 u_str ESTAB 0 0 /run/systemd/journal/stdout 6995 * 9451 u_dgr ESTAB 0 0 * 1806086 * 1806085 u_str ESTAB 0 0 * 13479 * 13480 u_dgr ESTAB 0 0 * 4829 * 4828 u_str ESTAB 0 0 * 3261568 * 3260817 u_dgr ESTAB 0 0 * 11586 * 5236 u_dgr ESTAB 0 0 * 7352 * 3543 u_str ESTAB 0 0 * 11647 * 11648 u_str ESTAB 0 0 * 7891 * 7892 u_dgr ESTAB 0 0 /run/systemd/notify 3543 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258766 * 3259810 u_str ESTAB 0 0 /run/dbus/system_bus_socket 12873 * 12872 u_str ESTAB 0 0 * 10290 * 10291 u_dgr ESTAB 0 0 * 1806088 * 1806087 u_str ESTAB 0 0 * 11625 * 11624 u_dgr ESTAB 0 0 * 12335 * 5236 u_str ESTAB 0 0 * 11651 * 11650 u_dgr ESTAB 0 0 /run/systemd/journal/dev-log 5236 * 0 u_dgr ESTAB 0 0 /run/systemd/journal/socket 5237 * 0 u_dgr ESTAB 0 0 * 3546 * 3547 u_str ESTAB 0 0 * 11632 * 11631 u_str ESTAB 0 0 * 11648 * 11647 u_str ESTAB 0 0 * 6974 * 6973 u_str ESTAB 0 0 * 12111 * 0 u_str ESTAB 0 0 * 3259676 * 3258625 u_str ESTAB 0 0 * 11675 * 11674 u_str ESTAB 0 0 * 11674 * 11675 u_str ESTAB 0 0 /run/systemd/journal/stdout 9419 * 9418 u_str ESTAB 0 0 * 11622 * 11621 u_dgr ESTAB 0 0 * 10127 * 10128 u_str ESTAB 0 0 * 7908 * 7019 u_str ESTAB 0 0 * 3259922 * 3259923 u_str ESTAB 0 0 * 3259921 * 3259920 u_str ESTAB 0 0 /run/systemd/journal/stdout 11068 * 13495 u_dgr ESTAB 0 0 * 10469 * 5236 u_dgr ESTAB 0 0 * 5032 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 1805048 * 1806090 u_str ESTAB 0 0 * 1805720 * 0 u_str ESTAB 0 0 * 10298 * 10299 u_str ESTAB 0 0 * 11689 * 11690 u_dgr ESTAB 0 0 * 1806087 * 1806088 u_str ESTAB 0 0 * 11290 * 11291 u_dgr ESTAB 0 0 * 11701 * 5236 u_str ESTAB 0 0 * 11693 * 11692 u_str ESTAB 0 0 * 13495 * 11068 u_dgr ESTAB 0 0 * 7081 * 5236 u_str ESTAB 0 0 * 11636 * 11635 u_str ESTAB 0 0 * 8975 * 9386 u_str ESTAB 0 0 @/run/rpcbind.sock 1805023 * 1805767 u_str ESTAB 0 0 /run/systemd/journal/stdout 9214 * 9661 u_dgr ESTAB 0 0 * 1806069 * 5237 u_str ESTAB 0 0 * 11621 * 11622 u_dgr ESTAB 0 0 /run/chrony/chronyd.sock 7086 * 0 u_str ESTAB 0 0 * 11680 * 11681 u_dgr ESTAB 0 0 * 3261581 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10159 * 10158 u_dgr ESTAB 0 0 * 6594 * 6593 u_str ESTAB 0 0 /var/lib/sss/pipes/pam 3260236 * 3261580 u_str ESTAB 0 0 * 10789 * 10790 u_str ESTAB 0 0 * 9451 * 6995 u_str ESTAB 0 0 * 11678 * 11677 u_str ESTAB 0 0 * 11668 * 11669 u_dgr ESTAB 0 0 * 6589 * 5237 u_str ESTAB 0 0 * 11644 * 11645 u_str ESTAB 0 0 * 11642 * 11641 u_dgr ESTAB 0 768 * 9007 * 9006 u_str ESTAB 0 0 * 1806581 * 1806580 u_str ESTAB 0 0 * 1806580 * 1806581 u_str ESTAB 0 0 /run/systemd/journal/stdout 10293 * 10292 u_str ESTAB 0 0 /run/systemd/journal/stdout 10291 * 10290 u_str ESTAB 0 0 * 3261597 * 3261598 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12063 * 13475 u_str ESTAB 0 0 * 11662 * 11663 u_str ESTAB 0 0 * 7035 * 7914 u_str ESTAB 0 0 /run/systemd/journal/stdout 7454 * 5011 u_str ESTAB 0 0 * 1805796 * 0 u_str ESTAB 0 0 * 3260520 * 0 u_str ESTAB 0 0 * 11657 * 11656 u_dgr ESTAB 0 0 * 1806083 * 1806084 u_str ESTAB 0 0 * 12624 * 12623 u_str ESTAB 0 0 * 9418 * 9419 u_dgr ESTAB 0 0 * 3258338 * 5237 u_str ESTAB 0 0 * 1805715 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 10299 * 10298 u_str ESTAB 0 0 * 11695 * 11696 u_str ESTAB 0 0 * 11681 * 11680 u_str ESTAB 0 0 * 3259838 * 0 u_str ESTAB 0 0 * 11631 * 11632 u_dgr ESTAB 0 0 * 12856 * 5237 u_str ESTAB 0 0 * 11653 * 11654 u_str ESTAB 0 0 /run/systemd/journal/stdout 9386 * 8975 u_str ESTAB 0 0 * 11665 * 11666 u_str ESTAB 0 0 * 9489 * 7097 u_dgr ESTAB 0 0 * 4828 * 4829 u_dgr ESTAB 0 0 * 12905 * 5236 u_str ESTAB 0 0 * 13475 * 12063 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6990 * 7893 u_str ESTAB 0 0 * 11650 * 11651 u_str ESTAB 0 0 * 9661 * 9214 u_str ESTAB 0 0 * 3258205 * 0 u_str ESTAB 0 0 * 11669 * 11668 u_str ESTAB 0 0 * 11294 * 11295 u_dgr ESTAB 0 0 * 9089 * 5236 u_str ESTAB 0 0 * 6577 * 7622 u_str ESTAB 0 0 * 3259920 * 3259921 u_dgr ESTAB 0 0 * 1806577 * 5236 u_str ESTAB 0 0 * 12623 * 12624 u_str ESTAB 0 0 * 11656 * 11657 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258625 * 3259676 u_str ESTAB 0 0 * 11686 * 11687 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11291 * 11290 u_str ESTAB 0 0 * 10790 * 10789 u_dgr ESTAB 0 0 * 9427 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6988 * 9429 u_str ESTAB 0 0 * 11683 * 11684 u_str ESTAB 0 0 * 3259923 * 3259922 u_str ESTAB 0 0 * 3259919 * 3259918 u_dgr ESTAB 0 0 * 3260669 * 5237 u_dgr ESTAB 0 0 * 1806061 * 5236 u_str ESTAB 0 0 /run/systemd/journal/stdout 11079 * 13510 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 13480 * 13479 u_str ESTAB 0 0 * 11639 * 11638 u_str ESTAB 0 0 * 11635 * 11636 u_dgr ESTAB 0 0 * 3544 * 3545 u_str ESTAB 0 0 * 11690 * 11689 u_str ESTAB 0 0 * 11672 * 11671 u_str ESTAB 0 0 /run/systemd/journal/stdout 11125 * 12904 u_dgr ESTAB 0 0 * 1806085 * 1806086 u_dgr ESTAB 0 0 * 3547 * 3546 u_str ESTAB 0 0 * 11696 * 11695 u_str ESTAB 0 0 * 3259810 * 3258766 u_dgr ESTAB 0 0 * 1795630 * 5236 u_str ESTAB 0 0 * 11654 * 11653 u_str ESTAB 0 0 * 1805767 * 1805023 u_str ESTAB 0 0 * 10688 * 10121 u_str ESTAB 0 0 * 11687 * 11686 u_str ESTAB 0 0 * 11671 * 11672 u_str ESTAB 0 0 * 12838 * 12062 u_str ESTAB 0 0 * 11629 * 11628 u_str ESTAB 0 0 * 9524 * 7991 u_str ESTAB 0 0 * 7892 * 7891 u_str ESTAB 0 0 * 11659 * 11660 u_str ESTAB 0 0 * 11645 * 11644 u_dgr ESTAB 0 0 * 9706 * 5236 u_dgr ESTAB 0 0 * 9006 * 9007 u_str ESTAB 0 0 * 10158 * 10159 u_dgr ESTAB 0 0 * 6972 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10121 * 10688 u_str ESTAB 0 0 * 1806041 * 1805036 u_str ESTAB 0 0 /run/systemd/journal/stdout 9213 * 7140 u_str ESTAB 0 0 /run/dbus/system_bus_socket 7097 * 9489 u_str ESTAB 0 0 /run/systemd/journal/stdout 7019 * 7908 u_str ESTAB 0 0 * 6973 * 6974 u_str ESTAB 0 0 * 11684 * 11683 u_str ESTAB 0 0 * 11663 * 11662 u_str ESTAB 0 0 * 1795634 * 0 u_dgr ESTAB 0 0 * 3260114 * 5237 u_str ESTAB 0 0 * 3261598 * 3261597 u_str ESTAB 0 0 * 3259918 * 3259919 u_str ESTAB 0 0 /run/systemd/journal/stdout 1805036 * 1806041 u_str ESTAB 0 0 * 11628 * 11629 u_str ESTAB 0 0 * 7140 * 9213 u_str ESTAB 0 0 * 11677 * 11678 u_str ESTAB 0 0 @f5c12eea9342a377/bus/systemd-logind/system 12872 * 12873 u_str ESTAB 0 0 @ed019b0aea426515/bus/systemd/bus-system 1806090 * 1805048 u_str ESTAB 0 0 @3cbcd21433f31955/bus/dbus-broker-lau/system 7893 * 6990 u_str ESTAB 0 0 @8aa9cf87f9fa122a/bus/systemd/bus-api-system 9429 * 6988 icmp6 UNCONN 0 0 *:ipv6-icmp *:* tcp ESTAB 0 0 <obfuscated>:952 <obfuscated>:nfs tcp ESTAB 0 36 <obfuscated>:ssh <obfuscated>:53277 tcp ESTAB 0 0 <obfuscated>:43916 <obfuscated>:ldaps tcp ESTAB 0 0 <obfuscated>:54296 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:40380 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:33806 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:39428 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:58586 <obfuscated>:30514 tcp ESTAB 0 0 [::1]:50086 [::1]:redis tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35174 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:50136 tcp ESTAB 0 0 [::1]:35174 [::1]:pcsync-https tcp CLOSE-WAIT 401224 0 [::1]:42184 [::1]:9004 tcp CLOSE-WAIT 339569 0 [::1]:34522 [::1]:9004 tcp CLOSE-WAIT 401329 0 [::1]:47524 [::1]:9004 tcp ESTAB 0 0 [::1]:redis [::1]:50086 tcp CLOSE-WAIT 339562 0 [::1]:56834 [::1]:9004 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:54998 tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35184 tcp ESTAB 0 0 [::1]:35184 [::1]:pcsync-https tcp FIN-WAIT-2 0 0 [::1]:9004 [::1]:42728 tcp CLOSE-WAIT 339529 0 [::1]:33512 [::1]:9004 tcp CLOSE-WAIT 401208 0 [::1]:42728 [::1]:9004After latest
dnf upgradepost service restart, and subsequent reboot for new kernel$ cat /proc/net/sockstat sockets: used 339 TCP: inuse 24 orphan 0 tw 36 alloc 47 mem 238 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294 -
Troubleshooting "TCP: out of memory" - Possible memory leak?
Hello
This is an early start of a thread for documenting a potential issue. I will add to it with findings accordinly.
I'm having some issues with on XOCE instance, commit
7a187. Three times this week, it has gotten quite unresponsive with network trafic. And looking at the console give clues as to why:
"TCP: out of memory -- consider tuning tcp_mem"
The weird part is that this is my most beefy XOCE instance of all.
4 cores, 8GiB, management agent 10 (Citrix), Rocky 10.2.In troubleshooting I tried restarting the XO-server service. And this seems to have resolved the issue for now. But TBH, I was looking at several point at the same time. So it is possible that it had resolved itself (unlikely but possible) whilst I was troubleshooting and reading up on
tcp_mem.My other XOCE instances are setup identically, and are actually a little bit smaller in the amount of RAM assigned. And they show no problems.
Failing XO instance connects to three separate 1Host pools.
Working XO instance connects to 1 HA-pool with three hosts.Installation is done by a port of https://github.com/cloudrootab/ansible_role_xoce, adjusted for Rocky10 instead of Ubuntu/Debian.
Edit: I should also add that this XOCE has shown the same behaviour on both a recently updated XCP-ng host, and one that is two months behind on updates. Running on Dell Inc. PowerEdge R670 for each of the hosts (RAM on hosts has plenty to spare, roughly 50% utilization)
-
RE: " can't compute delta" & "can't connect through NBD, fall back to stream export" after 2026-07-28
I'm seing the same behaviour as well.
To answer the questions you're asking for:XO from source (7a187)
XO is connected to three separate pools. Backups work most of the times. But I've started seeing the same error. However, it seems intermitent. And somwhat random. Since it will generate an error from one VM, but not on another on the same host with the same SR etc etc.
can't compute delta OpaqueRef: <ref> from OpaqueRef:<ref>, fall back to a full can't connect through NBD, fall back to stream export Backup fell back to a full- There is an assigned backup-network, and it is the same as the management network. With NBD enabled
- XO can access all of the hosts on the management/backup network
- IPv4
-
RE: (Windows) guest IPv6 address doesn't collapse zeroes -> Long IPv6 addresses
I've taken a quick look, looks like it'll be solved as part of the Windows guest agent overhaul, so please look forward to that.
Thanks for the info. I will be looking forward to that, indeed.
-
RE: Ubuntu 24.04 VMs not reporting IP addresses to XCP-NG 8.2.1
Unfortunately the Rust-based agent is effectively not progressing. Last commit was months ago, and the 0.4.0-release was over 2 years ago, Issues aren't tagged for many months. Simple and easy MR's are left hanging in limbo.
I'm hoping they will show this project some more love and attention soon. Since I really like the idea of an open source Rust-based agent. As it is clear other things are taking precedent over it.
-
RE: [Solved] SR_SOURCE_SPACE_INSUFFICIENT - Problems enabling HA
Thanks again for your input and recomendations! I'll verify that this is solved by having the LUN expanded to 8GB instead. Afterwards I'll mark your answer as the solution!
-
RE: [Solved] SR_SOURCE_SPACE_INSUFFICIENT - Problems enabling HA
Thank you for your reply.
The reasoning here is that we are experimenting quite heavily at this moment.
And the thinking here is to have three LUNs each for their intended purpose.LUNs
1 for VHD-based VDIs
1 for qcow2 based VDIs
The ones above will be used and modified quite a bit
1 for the Heartbeat. Being left alone to be as standard out-of-the-box as possible.All of the LUNs reside on the same storage systems (Dell PowerStore at the moment). So my resoning here is that they're all on the same storage cluster, and therefore will be affected similarily regardless. Exotic corner-cases may of course show up.
But I will absolutely take your recommendations into account! And when we've stopped messing with the storage for the VMs. I will have the Heartbeat there on that as well!