XOA - Console freezing every few seconds consistently.

trobertson

Hey everyone, long time XCP-NG user but also lurker.

I have 3 servers with XCP-NG 8.2 LTS, and have used since the 7.x days. I switched from XCP-NG Center to XOA some time back and eventually an odd issue appeared that I've been unable to diagnose or had the time to dedicate to.

When using XOA to access a VM and view from the Console tab, after 5-15 seconds the Console view just freezes. If I click off to the Network tab or elsewhere and click back, it will work again momentarily.

Doesn't matter which server I connect to, which VM I connect to, what browser or machine I connect from. I've also built XOA from source and recently recreated it as a docker container. I've also tried from my reverse proxy an directly via IP, all to the same results.

All my configurations are currently up to date for 8.2 LTS and even a recently installed XCP-NG 8.2 LTS with latest ISO on an Dell R730 shows the same issue. I'm sure it's something silly but I definitely haven't been able to figure it out. It's not a show stopper as I use other remote means and even XCP-NG Center's console still works without issue.

Appreciate it all!

olivierlambert

Hi,

What kind of NICs do you have?

edit: that's interesting about consoles working on XenCenter. Have you tried XO Lite?

trobertson

Hey Oliver! I've followed a lot of your advice over the years.

I have 3 servers, and 3 separate pools (other reasons). Each server has a dedicated management eth connection without VLAN tags and separate SFP/DACs for the VMs.

Dell T620
Intel Corporation I350 Gigabit Network Connection | x4 @ 1Gbps (Management Eth)
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ | x1 @ 10Gbps (VLAN)

Dell R730
Intel Corporation I350 Gigabit Network Connection | x2 @ 1Gbps (Management Eth)
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) | x2 @ 10Gbps (VLAN)

Dell R730XD
Intel Corporation I350 Gigabit Network Connection | x2 @ 1Gbps (Management Eth)
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) | x2 @ 10Gbps (VLAN)

XO-Lite was just tested and there is no freezing from XO-Lite's console

olivierlambert

That's interesting. Can you try with /v6 in your XOA URL to see if you still have a console issue?

trobertson

Hey Oliver, apologies, this is an item I'm not aware of.

Where should I add /v6 into the URL?

trobertson

Ah apologies, looking at other comments, I am built from Source without a license atm. I don't believe I get the preview

olivierlambert

See https://xen-orchestra.com/blog/xen-orchestra-5-98#️-xo-6

To access the new preview UI, just add /v6 in your XOA URL. If you are using it from the sources, you can build it with yarn run turbo run build --filter @xen-orchestra/web from the XO root folder.

trobertson

Ah, I see now. I was adding it to the wrong location of the URL. I did load it up and it did freeze

Currently I have the same VM up in all 3 consoles (XO, XOv6, XO Lite)

XO and XOv6 both froze at differing times
XO lite is still going.

I've tried having XO and the host servers on the same VLAN, both tagged and untagged. I don't see any packet drops between them in monitoring.

trobertson

Well that did help me to find something I did not notice before! An oddity in which I see the XO webserver traffic between itself and the host is routing oddly over my networking, will try to trace down as to why.

The short and sweet is I see the XO TCP calls come across the wrong interface in routing to the Host at 443, since the traffic isn't expected in this direction it has no stateful tracking and it's denied. This doesn't seem to impact anything else except the console streams and likely a misconfiguration in my host setup I'm assuming.

Appreciate the help!

olivierlambert

Nice, that's interesting

trobertson

So I'm not 100% but I've resolved the symptom.

I have 3 VLANS involved

DATA - Endpoints
PRIV - Reverse proxy manager exists here
MGMT - Where XOA exists.

Myself and users access XOA via the reverse proxy, which access the webserver in the MGMT vlan.

My hosts had their primary management access on the DATA vlan, a holdover setup from prior, and a secondary management access on the MGMT VLAN and a third for data migrations.

XOA was connected to the MGMT IPs of each host.

Watching packet capture I see packets moving from the originating endpoint in DATA, to the reverse proxy, from it to the XOA backend server. Then suddenly I see XOA in the MGMT VLAN try to access the hosts directly in the DATA VLAN which was blocked.

Problem is, it seemed to send packets meant to originate from the MGMT-VLAN as sourced from the DATA-VLAN.

The resolution was I just removed the secondary management IP and switched the primary management of each host to the MGMT-VLAN alongside XOA.

They're happier now at least.

trobertson

Appreciate the assistance on it all. This is only a homelab turned private cloud setup, but XCP-NG has been a pleasure to work with, thanks for all you and the team do!

olivierlambert

Great news, nice catch