Hello,
Windows only supports adding CPUs but not removing them.
This is a Windows limitation.
I'm not surprised by that limitation. Thanks for confirming anyway.
Cheers!
J
Offline
Posts
-
RE: Reducing vCPU isn't done live on Windows
-
V2V migrated VMs don't autostart
I'm at the moment doing alot of test with V2V (applause on getting rid of the vddk-dependency btw
)However, what I'm noticing is that the migrated VMs don't automatically start when the import has completed. This is contrary to current documentation.
Is this a potential bug? Or am I missing something obvious?
PS: The imports are all sucessfull. It's just the auto-start part that I'm noticing doesn't work.
Cheers!
Commit: dc573
Template used for testig is: Rocky Linux 9
-
Reducing vCPU isn't done live on Windows
I'm doing a couple of tests to trying to figure out quirks and behaviours from XCP-ng side, for a future production (licensed) setup.
So I tried to see what would happen if I would play with the number of vCPUs on two guests.
TL;DR: Reducing vCPUs on Windows requires reboot.On Windows Guest:
OS: Windows Server 25
Guest Agent: 9.2.385.0
Increasing the number of vCPUs up to the max value, is done live and works well. But reducing the vCPUs requires a reboot.However. On a Rocky Linux 10 installation. The reducing works immediately live and online as expected.
This might very well be a limitation of how Windows does things internally. But I thought it interesting enouch to report about anyway.
-
RE: Why are transfer sizes different between XO5 & XO6?
fix/update-transfer-size-calculation
I can confirm that XO6 now shows the same transfer sizes as XO5. Lovely
-
RE: Why are transfer sizes different between XO5 & XO6?
@Danp
I have Healtch Checks on my Delta Backups. Yes -
RE: Why are transfer sizes different between XO5 & XO6?
Same thing here:
Commit: a1dbaXO5 transfers ~ 9-20GB
XO6 transfers ~ 100-180GB
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
I have now uploaded two heap-snapshots for you to look at.
If there is anything else, big or small, that I can help out with. Just let me knowThanks!
-
RE: (Windows) guest IPv6 address doesn't collapse zeroes -> Long IPv6 addresses
I just did a quick test to check. And it does indeed seem that this (small) issue has now been resolved.
Tested on Windows Server 25, Management agent 9.2.385-0.Cheers!
-
RE: Master, commit a3139 failing backups
I've just been hit with this also: Can't init vhd directory without using alias
All S3 repo's running SeaweedFS, where fine the last time I used them (a few weeks ago). I don't use my xcpng servers on the homelab daily as I move to Unraid.
Running Master, commit 99312
I'm seeing the same error, with the same reported commit.
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
I now have two heap snapshots. Could you provide somewhere to upload them, please? And I will do so as soon as I can.
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
@jr-m4 you can export the heap memory of the nodeJS process by doing
kill -SIGUSER2 <xoserverpid>onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment
Do you have somewhere I can upload the heapsnapshot? (148MB)
Ping @poddingue as well, for visibility
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
Looking at it today again. Doing a
sudo ss -tpand noticing that there are alot ofCLOSE-WAIT. Maybe this is a red herring, or maybe not. I thought I would share the findings regardless.The output is also to long for me to be able to paste it in text. So I've attached it as a txt-file.
close-wait.txt -
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising
tcp_membuys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on anxo_serverrestart points the same way.
It might be worth a mention to @Team-XO-Backend, since they would know whether
xo-serverkeeps sockets per pool.This is why I'm not increasing the numbers quite yet. Since this would only mask a potential issue. If an underlying cause is responsible, then this would be of interest.
Thanks for chiming in!
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
Did a much closer look and can indeed confirm that the network capabilities go up from kb/s -> MB/s immediately from xo_server restarts.
-
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
@Forza
Comparing against the weaker XOCEStrong:
net.ipv4.tcp_mem = 90147 120196 180294 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304Weak:
net.ipv4.tcp_mem = 41766 55688 83532 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 -
RE: Troubleshooting "TCP: out of memory" - Possible memory leak?
Adding the initial memstats I was looking at whilst trying to troubleshoot
$ cat /proc/net/sockstat sockets: used 3478 TCP: inuse 37 orphan 0 tw 25 alloc 3149 mem 194631 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294Output of
sudo ssNetid State Recv-Q Send-Q Local Address:Port Peer Address:Port u_str ESTAB 0 0 * 10292 * 10293 u_str ESTAB 0 0 /var/lib/sss/pipes/nss 3260817 * 3261568 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12062 * 12838 u_str ESTAB 0 0 * 11638 * 11639 u_str ESTAB 0 0 * 13510 * 11079 u_str ESTAB 0 0 * 11624 * 11625 u_dgr ESTAB 0 768 * 10128 * 10127 u_dgr ESTAB 0 0 * 6593 * 6594 u_str ESTAB 0 0 * 5011 * 7454 u_str ESTAB 0 0 * 11692 * 11693 u_str ESTAB 0 0 * 11641 * 11642 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11295 * 11294 u_dgr ESTAB 0 0 * 6309 * 5237 u_str ESTAB 0 0 * 3261580 * 3260236 u_str ESTAB 0 0 * 3259924 * 3259925 u_dgr ESTAB 0 0 * 1806084 * 1806083 u_str ESTAB 0 0 * 12904 * 11125 u_dgr ESTAB 0 0 * 3545 * 3544 u_str ESTAB 0 0 * 3259925 * 3259924 u_str ESTAB 0 0 * 11666 * 11665 u_str ESTAB 0 0 /run/systemd/journal/stdout 7622 * 6577 u_str ESTAB 0 0 * 11660 * 11659 u_str ESTAB 0 0 /run/systemd/journal/stdout 7991 * 9524 u_str ESTAB 0 0 * 1806062 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 7914 * 7035 u_str ESTAB 0 0 /run/systemd/journal/stdout 6995 * 9451 u_dgr ESTAB 0 0 * 1806086 * 1806085 u_str ESTAB 0 0 * 13479 * 13480 u_dgr ESTAB 0 0 * 4829 * 4828 u_str ESTAB 0 0 * 3261568 * 3260817 u_dgr ESTAB 0 0 * 11586 * 5236 u_dgr ESTAB 0 0 * 7352 * 3543 u_str ESTAB 0 0 * 11647 * 11648 u_str ESTAB 0 0 * 7891 * 7892 u_dgr ESTAB 0 0 /run/systemd/notify 3543 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258766 * 3259810 u_str ESTAB 0 0 /run/dbus/system_bus_socket 12873 * 12872 u_str ESTAB 0 0 * 10290 * 10291 u_dgr ESTAB 0 0 * 1806088 * 1806087 u_str ESTAB 0 0 * 11625 * 11624 u_dgr ESTAB 0 0 * 12335 * 5236 u_str ESTAB 0 0 * 11651 * 11650 u_dgr ESTAB 0 0 /run/systemd/journal/dev-log 5236 * 0 u_dgr ESTAB 0 0 /run/systemd/journal/socket 5237 * 0 u_dgr ESTAB 0 0 * 3546 * 3547 u_str ESTAB 0 0 * 11632 * 11631 u_str ESTAB 0 0 * 11648 * 11647 u_str ESTAB 0 0 * 6974 * 6973 u_str ESTAB 0 0 * 12111 * 0 u_str ESTAB 0 0 * 3259676 * 3258625 u_str ESTAB 0 0 * 11675 * 11674 u_str ESTAB 0 0 * 11674 * 11675 u_str ESTAB 0 0 /run/systemd/journal/stdout 9419 * 9418 u_str ESTAB 0 0 * 11622 * 11621 u_dgr ESTAB 0 0 * 10127 * 10128 u_str ESTAB 0 0 * 7908 * 7019 u_str ESTAB 0 0 * 3259922 * 3259923 u_str ESTAB 0 0 * 3259921 * 3259920 u_str ESTAB 0 0 /run/systemd/journal/stdout 11068 * 13495 u_dgr ESTAB 0 0 * 10469 * 5236 u_dgr ESTAB 0 0 * 5032 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 1805048 * 1806090 u_str ESTAB 0 0 * 1805720 * 0 u_str ESTAB 0 0 * 10298 * 10299 u_str ESTAB 0 0 * 11689 * 11690 u_dgr ESTAB 0 0 * 1806087 * 1806088 u_str ESTAB 0 0 * 11290 * 11291 u_dgr ESTAB 0 0 * 11701 * 5236 u_str ESTAB 0 0 * 11693 * 11692 u_str ESTAB 0 0 * 13495 * 11068 u_dgr ESTAB 0 0 * 7081 * 5236 u_str ESTAB 0 0 * 11636 * 11635 u_str ESTAB 0 0 * 8975 * 9386 u_str ESTAB 0 0 @/run/rpcbind.sock 1805023 * 1805767 u_str ESTAB 0 0 /run/systemd/journal/stdout 9214 * 9661 u_dgr ESTAB 0 0 * 1806069 * 5237 u_str ESTAB 0 0 * 11621 * 11622 u_dgr ESTAB 0 0 /run/chrony/chronyd.sock 7086 * 0 u_str ESTAB 0 0 * 11680 * 11681 u_dgr ESTAB 0 0 * 3261581 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10159 * 10158 u_dgr ESTAB 0 0 * 6594 * 6593 u_str ESTAB 0 0 /var/lib/sss/pipes/pam 3260236 * 3261580 u_str ESTAB 0 0 * 10789 * 10790 u_str ESTAB 0 0 * 9451 * 6995 u_str ESTAB 0 0 * 11678 * 11677 u_str ESTAB 0 0 * 11668 * 11669 u_dgr ESTAB 0 0 * 6589 * 5237 u_str ESTAB 0 0 * 11644 * 11645 u_str ESTAB 0 0 * 11642 * 11641 u_dgr ESTAB 0 768 * 9007 * 9006 u_str ESTAB 0 0 * 1806581 * 1806580 u_str ESTAB 0 0 * 1806580 * 1806581 u_str ESTAB 0 0 /run/systemd/journal/stdout 10293 * 10292 u_str ESTAB 0 0 /run/systemd/journal/stdout 10291 * 10290 u_str ESTAB 0 0 * 3261597 * 3261598 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12063 * 13475 u_str ESTAB 0 0 * 11662 * 11663 u_str ESTAB 0 0 * 7035 * 7914 u_str ESTAB 0 0 /run/systemd/journal/stdout 7454 * 5011 u_str ESTAB 0 0 * 1805796 * 0 u_str ESTAB 0 0 * 3260520 * 0 u_str ESTAB 0 0 * 11657 * 11656 u_dgr ESTAB 0 0 * 1806083 * 1806084 u_str ESTAB 0 0 * 12624 * 12623 u_str ESTAB 0 0 * 9418 * 9419 u_dgr ESTAB 0 0 * 3258338 * 5237 u_str ESTAB 0 0 * 1805715 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 10299 * 10298 u_str ESTAB 0 0 * 11695 * 11696 u_str ESTAB 0 0 * 11681 * 11680 u_str ESTAB 0 0 * 3259838 * 0 u_str ESTAB 0 0 * 11631 * 11632 u_dgr ESTAB 0 0 * 12856 * 5237 u_str ESTAB 0 0 * 11653 * 11654 u_str ESTAB 0 0 /run/systemd/journal/stdout 9386 * 8975 u_str ESTAB 0 0 * 11665 * 11666 u_str ESTAB 0 0 * 9489 * 7097 u_dgr ESTAB 0 0 * 4828 * 4829 u_dgr ESTAB 0 0 * 12905 * 5236 u_str ESTAB 0 0 * 13475 * 12063 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6990 * 7893 u_str ESTAB 0 0 * 11650 * 11651 u_str ESTAB 0 0 * 9661 * 9214 u_str ESTAB 0 0 * 3258205 * 0 u_str ESTAB 0 0 * 11669 * 11668 u_str ESTAB 0 0 * 11294 * 11295 u_dgr ESTAB 0 0 * 9089 * 5236 u_str ESTAB 0 0 * 6577 * 7622 u_str ESTAB 0 0 * 3259920 * 3259921 u_dgr ESTAB 0 0 * 1806577 * 5236 u_str ESTAB 0 0 * 12623 * 12624 u_str ESTAB 0 0 * 11656 * 11657 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258625 * 3259676 u_str ESTAB 0 0 * 11686 * 11687 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11291 * 11290 u_str ESTAB 0 0 * 10790 * 10789 u_dgr ESTAB 0 0 * 9427 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6988 * 9429 u_str ESTAB 0 0 * 11683 * 11684 u_str ESTAB 0 0 * 3259923 * 3259922 u_str ESTAB 0 0 * 3259919 * 3259918 u_dgr ESTAB 0 0 * 3260669 * 5237 u_dgr ESTAB 0 0 * 1806061 * 5236 u_str ESTAB 0 0 /run/systemd/journal/stdout 11079 * 13510 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 13480 * 13479 u_str ESTAB 0 0 * 11639 * 11638 u_str ESTAB 0 0 * 11635 * 11636 u_dgr ESTAB 0 0 * 3544 * 3545 u_str ESTAB 0 0 * 11690 * 11689 u_str ESTAB 0 0 * 11672 * 11671 u_str ESTAB 0 0 /run/systemd/journal/stdout 11125 * 12904 u_dgr ESTAB 0 0 * 1806085 * 1806086 u_dgr ESTAB 0 0 * 3547 * 3546 u_str ESTAB 0 0 * 11696 * 11695 u_str ESTAB 0 0 * 3259810 * 3258766 u_dgr ESTAB 0 0 * 1795630 * 5236 u_str ESTAB 0 0 * 11654 * 11653 u_str ESTAB 0 0 * 1805767 * 1805023 u_str ESTAB 0 0 * 10688 * 10121 u_str ESTAB 0 0 * 11687 * 11686 u_str ESTAB 0 0 * 11671 * 11672 u_str ESTAB 0 0 * 12838 * 12062 u_str ESTAB 0 0 * 11629 * 11628 u_str ESTAB 0 0 * 9524 * 7991 u_str ESTAB 0 0 * 7892 * 7891 u_str ESTAB 0 0 * 11659 * 11660 u_str ESTAB 0 0 * 11645 * 11644 u_dgr ESTAB 0 0 * 9706 * 5236 u_dgr ESTAB 0 0 * 9006 * 9007 u_str ESTAB 0 0 * 10158 * 10159 u_dgr ESTAB 0 0 * 6972 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10121 * 10688 u_str ESTAB 0 0 * 1806041 * 1805036 u_str ESTAB 0 0 /run/systemd/journal/stdout 9213 * 7140 u_str ESTAB 0 0 /run/dbus/system_bus_socket 7097 * 9489 u_str ESTAB 0 0 /run/systemd/journal/stdout 7019 * 7908 u_str ESTAB 0 0 * 6973 * 6974 u_str ESTAB 0 0 * 11684 * 11683 u_str ESTAB 0 0 * 11663 * 11662 u_str ESTAB 0 0 * 1795634 * 0 u_dgr ESTAB 0 0 * 3260114 * 5237 u_str ESTAB 0 0 * 3261598 * 3261597 u_str ESTAB 0 0 * 3259918 * 3259919 u_str ESTAB 0 0 /run/systemd/journal/stdout 1805036 * 1806041 u_str ESTAB 0 0 * 11628 * 11629 u_str ESTAB 0 0 * 7140 * 9213 u_str ESTAB 0 0 * 11677 * 11678 u_str ESTAB 0 0 @f5c12eea9342a377/bus/systemd-logind/system 12872 * 12873 u_str ESTAB 0 0 @ed019b0aea426515/bus/systemd/bus-system 1806090 * 1805048 u_str ESTAB 0 0 @3cbcd21433f31955/bus/dbus-broker-lau/system 7893 * 6990 u_str ESTAB 0 0 @8aa9cf87f9fa122a/bus/systemd/bus-api-system 9429 * 6988 icmp6 UNCONN 0 0 *:ipv6-icmp *:* tcp ESTAB 0 0 <obfuscated>:952 <obfuscated>:nfs tcp ESTAB 0 36 <obfuscated>:ssh <obfuscated>:53277 tcp ESTAB 0 0 <obfuscated>:43916 <obfuscated>:ldaps tcp ESTAB 0 0 <obfuscated>:54296 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:40380 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:33806 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:39428 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:58586 <obfuscated>:30514 tcp ESTAB 0 0 [::1]:50086 [::1]:redis tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35174 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:50136 tcp ESTAB 0 0 [::1]:35174 [::1]:pcsync-https tcp CLOSE-WAIT 401224 0 [::1]:42184 [::1]:9004 tcp CLOSE-WAIT 339569 0 [::1]:34522 [::1]:9004 tcp CLOSE-WAIT 401329 0 [::1]:47524 [::1]:9004 tcp ESTAB 0 0 [::1]:redis [::1]:50086 tcp CLOSE-WAIT 339562 0 [::1]:56834 [::1]:9004 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:54998 tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35184 tcp ESTAB 0 0 [::1]:35184 [::1]:pcsync-https tcp FIN-WAIT-2 0 0 [::1]:9004 [::1]:42728 tcp CLOSE-WAIT 339529 0 [::1]:33512 [::1]:9004 tcp CLOSE-WAIT 401208 0 [::1]:42728 [::1]:9004After latest
dnf upgradepost service restart, and subsequent reboot for new kernel$ cat /proc/net/sockstat sockets: used 339 TCP: inuse 24 orphan 0 tw 36 alloc 47 mem 238 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294 -
Troubleshooting "TCP: out of memory" - Possible memory leak?
Hello
This is an early start of a thread for documenting a potential issue. I will add to it with findings accordinly.
I'm having some issues with on XOCE instance, commit
7a187. Three times this week, it has gotten quite unresponsive with network trafic. And looking at the console give clues as to why:
"TCP: out of memory -- consider tuning tcp_mem"
The weird part is that this is my most beefy XOCE instance of all.
4 cores, 8GiB, management agent 10 (Citrix), Rocky 10.2.In troubleshooting I tried restarting the XO-server service. And this seems to have resolved the issue for now. But TBH, I was looking at several point at the same time. So it is possible that it had resolved itself (unlikely but possible) whilst I was troubleshooting and reading up on
tcp_mem.My other XOCE instances are setup identically, and are actually a little bit smaller in the amount of RAM assigned. And they show no problems.
Failing XO instance connects to three separate 1Host pools.
Working XO instance connects to 1 HA-pool with three hosts.Installation is done by a port of https://github.com/cloudrootab/ansible_role_xoce, adjusted for Rocky10 instead of Ubuntu/Debian.
Edit: I should also add that this XOCE has shown the same behaviour on both a recently updated XCP-ng host, and one that is two months behind on updates. Running on Dell Inc. PowerEdge R670 for each of the hosts (RAM on hosts has plenty to spare, roughly 50% utilization)
-
RE: " can't compute delta" & "can't connect through NBD, fall back to stream export" after 2026-07-28
I'm seing the same behaviour as well.
To answer the questions you're asking for:XO from source (7a187)
XO is connected to three separate pools. Backups work most of the times. But I've started seeing the same error. However, it seems intermitent. And somwhat random. Since it will generate an error from one VM, but not on another on the same host with the same SR etc etc.
can't compute delta OpaqueRef: <ref> from OpaqueRef:<ref>, fall back to a full can't connect through NBD, fall back to stream export Backup fell back to a full- There is an assigned backup-network, and it is the same as the management network. With NBD enabled
- XO can access all of the hosts on the management/backup network
- IPv4
-
RE: (Windows) guest IPv6 address doesn't collapse zeroes -> Long IPv6 addresses
I've taken a quick look, looks like it'll be solved as part of the Windows guest agent overhaul, so please look forward to that.
Thanks for the info. I will be looking forward to that, indeed.