Broken Host Consoles in fresh xcp-ng 8 installation. VM consoles work.

rjt

@olivierlambert
No NAT. Same subnet.
xcp-ng 7 host consoles work fine in same xcp center instance.
Fresh install, reboot, and yum update and rebooted.
I will disable physical GPUs via disabling their PCI slots and report back.

rjt

@woodguy908 Do you happen to have any dedicated graphic cards in your host? I threw old desktop cards with physical outputs in for a test. But this was not a problem earlier. The yum update is likely source of the problem.

woodguy908

@rjt No - I don't have a GPU installed on that server.

olivierlambert

Hmm strange… I can't reproduce the issue. Anyone else with the problem?

Can you share your server specs so we can try to find a pattern?

rjt

@olivierlambert Are you using a Linux client? Wasnt there a recent patch to fix redrawing the screen on Linux clients?

olivierlambert

I'm not sure to understand the connection with this issue? Regardless the OS/client, I don't have the issue on Xen Orchestra.

rjt

[01:05 xcp-ng-G ~]# xe console-list
uuid ( RO)             : fb69931e-ae2b-c2d3-dbce-3e9a9ae57646
          vm-uuid ( RO): 2149b4df-f113-42dd-8ac5-4eff82ac1b0b
    vm-name-label ( RO): Control domain on host: xcp-ng-G
         protocol ( RO): VT100
         location ( RO): https://192.168.10.192/console?ref=OpaqueRef:8159b2a9-8e7b-450c-ac1c-2d6ac7d64e84

uuid ( RO)             : 12aae74a-4d4f-7ed3-976d-b36e0ae1905d
          vm-uuid ( RO): 2149b4df-f113-42dd-8ac5-4eff82ac1b0b
    vm-name-label ( RO): Control domain on host: xcp-ng-G
         protocol ( RO): RFB
         location ( RO): https://192.168.10.192/console?ref=OpaqueRef:71434657-a37c-4ba2-bdb1-64d8f45a1a3c

[01:07 xcp-ng-G ~]# ip a show | egrep inet | egrep -v '(inet 127)'
    inet 192.168.10.193/16 brd 192.168.255.255 scope global xenbr0

192.168.10.193 != 192.168.10.192 Control domain consoles are still using a very very old IP address totally different than the statically leased DHCP address of Dom0. Even after an emergency network reset. Cannot delete this VM. The location parameter is ReadOnly, so i cannot delete it. I am going to try another Emergency Network Reset and set the static lease as a static IP address. If that does not work, is there a way to blow away these bad consoles or at least the IP addresses?

Because i knew i would wipe the machine and start over fresh anyway, i tried to save time by using DHCP.

rjt

Example from another machine on which the IP address of the Control domain VM does not match the actual IP address of Dom0 192.168.2.141 != 192.168.10.192:

[01:23 eceoxen-B ~]# xe console-list vm-name-label=Control\ domain\ on\ host:\ eceoxen-B
uuid ( RO)             : d5039d1a-64ad-c8a9-a309-51e568ba2926
          vm-uuid ( RO): 1593da28-8e85-4252-878e-778eb414c549
    vm-name-label ( RO): Control domain on host: eceoxen-B
         protocol ( RO): VT100
         location ( RO): https://192.168.2.141/console?ref=OpaqueRef:7be93ca1-76cf-4649-8276-74891eac0a06

uuid ( RO)             : 7bcf725e-ae83-cf6f-7997-7dd63469929f
          vm-uuid ( RO): 1593da28-8e85-4252-878e-778eb414c549
    vm-name-label ( RO): Control domain on host: eceoxen-B
         protocol ( RO): RFB
         location ( RO): https://192.168.2.141/console?ref=OpaqueRef:1361b935-2d25-4929-9b27-4b0483cbb0f7

[01:25 eceoxen-B ~]# ip a show  | egrep inet | egrep -v '(inet 127)'
    inet 192.168.10.192/16 brd 192.168.255.255 scope global dynamic xenbr0

rjt

Attempts to reset or clear the Dom0 console entries ....

[02:37 xcp-ng-G ~]# xe console-param-clear param-name=location uuid=fb69931e-ae2b-c2d3-dbce-3e9a9ae57646
Error: Can only clear RW parameters

[02:38 xcp-ng-G ~]# xe console-param-clear uuid=12aae74a-4d4f-7ed3-976d-b36e0ae1905d param-name=location
Error: Can only clear RW parameters

[02:38 xcp-ng-G ~]# xe console-param-remove uuid=12aae74a-4d4f-7ed3-976d-b36e0ae1905d param-key=location param-name=location
Error: Can only remove from parameters of type Set or Map

[02:38 xcp-ng-G ~]# xe console-param-set uuid=12aae74a-4d4f-7ed3-976d-b36e0ae1905d

Is there some way to blow these old Dom0 consoles away?

olivierlambert

Does a reboot fix anything? Try to see if you have network parameters set somewhere. Maybe XCP-ng Center saved some config?

rjt

I have rebooted the hosts numerous times. xcp center configuration looks good.
Tried reapplying network config, but nothing.
Tried the xsconsole emergency network reset.
Tried the firstboot service and /etc/firstboot.d/, but no luck.
Wonder if it is picking up configuration information from another harddrive used to boot xcp.
Resigned to wipe and reinstall.

olivierlambert

I don't know how this thing might be stuck in your config. At worst you can edit XAPI DB manually and change the value.

Stop XAPI service on all hosts of the pool
Copy /var/xapi/state.db somewhere else in case
Edit it, and find the bad IP, replace it with the right one
Save
Start XAPI on all hosts, master first

rjt

@olivierlambert
The /var/lib/xcp/state.db is basically a single line of xml with a half-million characters.
The old egrep or sed or vi searching did not seem to narrow things down likely they normally do. Used xmllint to put the db in a pretty format with line breaks and indentation. Then vi to edit. Of course, as soon as i start xapi, those changes are lost. I assume there must be a massive speedup in lookups when all on a single line.

[23:59 xen-B xcp]# pushd /var/lib/xcp/
[23:59 xen-B xcp]# systemctl stop xapi
[23:59 xen-B xcp]# cp state.db ./state-YYYYMMDD-HHMM.db
[23:59 xen-B xcp]# egrep  '(192.168.2.141)' ./state.db | wc
      1   11757  454034
[23:59 xen-B xcp]# echo " :( 1 line with over 11,000 words :(" 
[23:59 xen-B xcp]# xmllint --format state.db >> state.xmllint--pretty.db
[23:59 xen-B xcp]# egrep  '(192.168.2.141)' ./state.db | wc
     10     193   13526
[23:59 xen-B xcp]# echo "i can deal with 193 words :) and edited with vi" 
[23:59 xen-B xcp]# mv state.xmllint--pretty.db ./state.db
[23:59 xen-B xcp]# systemctl start xapi

xe console-list indicates the the proper IP addresses:

[23:59 xen-B xcp]# xe console-list  vm-name-label=Control\ domain\ on\ host:\ xen-B

uuid ( RO)             : d5039d1a-64ad-c8a9-a309-51e568ba2926
          vm-uuid ( RO): 1593da28-8e85-4252-878e-778eb414c549
    vm-name-label ( RO): Control domain on host: xen-B
         protocol ( RO): VT100
         location ( RO): https://192.168.10.192/console?ref=OpaqueRef:7be93ca1-76cf-4649-8276-74891eac0a06


uuid ( RO)             : 7bcf725e-ae83-cf6f-7997-7dd63469929f
          vm-uuid ( RO): 1593da28-8e85-4252-878e-778eb414c549
    vm-name-label ( RO): Control domain on host: xen-B
         protocol ( RO): RFB
         location ( RO): https://192.168.10.192/console?ref=OpaqueRef:1361b935-2d25-4929-9b27-4b0483cbb0f7


[00:00 xen-B xcp]# ip a show dev xenbr0 | egrep inet
    inet 192.168.10.192/16 brd 192.168.255.255 scope global dynamic xenbr0

But there is still a blank host console in both XOA and xcp center. Suppose those OpaqueRefs have to be fixed up as well. EFI starting crashing on this r720, so having many other severe issues to deal with.

rjt

Anybody have an idea of what to do to get our consoles back?

borzel

Do a recursive grep in /etc to find all files that have the old IP address

Like grep -R 'IP-Address' /etc

rjt

@borzel i had tried that but found grep did not work like i am used. egrep worked better, but still not like how i am used to..

 egrep -R '(10\.40\.|192\.168\.)'  /etc

DreDay

I've seen that before where when you initially set the management IP via DHCP it breaks the console when the IP changes. XCP-ng does seem to register that the IP has changed and so it does not update the console URI.

We were able to resolve the issue by setting the IP statically on the XCP-ng node(I believe a reboot was also required). That updated the console's URI and it was working again in XCP-ng Center and XO.

rjt

@DreDay yes, statically leased DHCP addresses that used to work on spinning platters of rust no longer work on Solid State Drives.

However, setting static IP address did not work to fix it. Had to reinstall, setting a static IP at re-installation time.

asdfonseca

@olivierlambert , i have the same problem, but only under NAT. In same subnet, both host and VM consoles work properly.

There's any special configuration to work under NAT?

olivierlambert

When XO will ask to access host console, XCP-ng XAPI will return the local host console IP (in the same network).

So if you XO can't reach this IP, console connection can't work.