Unable to Access MGMT interface/ No NICS detected
-
So TLDR other day my server shut off from storms, and it refused to come back up due to no NIC's / IP detected. Did a whole bunch of commands, network reset, xcp network reset, and still no dice.
Detailed version.
R720XD everything has been running fine for months until the other day a storm came in and knocked everything off line ( i guess my 1500va 1000W UPS isnt enough for a R720XD nearly idle lol )
99% of everything came back up no problems minus this server, went to see it said no MGMT interface/IP detected. o okay cool no problem ill just go in and give it the ip again and boom money right? nope,This server on the Switch/UCG Fiber side shows a IP of 10.0.100.26, and using IP a I see the same thing. Cool we are solid, Nope, Still no dice.
So then me and GPT started to go ham on running commands, reseting the network, checking XAPI status, the workssss And still no dice.
When Running OVS-VSCTL SHOW
I see the bridge Xenbr0, i see the port, the interface, the type, and then i also see eth3 interface.
And then i went inside the network-scripts and checked the ifcfg-eth3 and ifcfg-xenbr0 and added those in since it was empty, saved and exited, ran SYSTEMCTL RESTART NETWORK and SYSTEMCTL RESTART XAPI, AND STATUS, and came back active and running Ran XE PIF-list Error
So i ran Through did these commands
xe-toolstack-restart
systemctl restart xapi
Then pinged 8.8.8.8 and it did fine, so i thought i was good
but hit a error on xe host-listthen i ran
ps aux | grep xapi
Then this is where i gave up, because everything i tried with GPT and my limited knowledge yielded no results.I have no issue Reinstalling XCP-NG buttttt 1 VM has some crucial data that i need to get pulled off before i wipe and Reinstall.
Im currently looking for a secondary NIC to see if maybeeee my onboard NIC is dying out.
IIRC Im on Version 8.3
Is there a command or something i can run that validates and repairs files or anything like that?
-
You need to take a look at XAPI log to check why XAPI can't start. It might help to pinpoint the problem. Also doing a
dmesgto watch about anything suspicious is a good practice. -
@olivierlambert is there anything in particular I need to keep a eye /look for? Or maybe pull the logs in smaller sections since im having to Idrac into the machine. It wont let me scroll the logs
-
Anything with "error" might be interesting, kernel oops or whatever network driver problem.
-
@olivierlambert Ran dmesg and its showing a WHOLEEEE bunch but i cant see up past about 40 lines up, and either Idrac, Or the Shell wont let me scroll up.
Running This command yields me a metric tonnnn of information ( last 100 Lines )
tail -f /var/log/xensource.log
and its not letting me scroll up or down to see much more what the screen shows sadly.
-
Gut feeling while watching your dmesg log: your Broadcom NIC is in "hybrid mode" between FC and ethernet. That might trigger your problem somehow. I would flash it to get in ethernet only. Check the exact NIC you have to see how to do this.
-
@olivierlambert I'll give that a shot and see! The NIC ive been using has 2 sfp ports and 2 ethernet ports. ( on board ) and I found a new nic laying around just ethernets I dropped in last night. That imma see what happens when I get home.
If it dont work ill flash the on board sfp/ethernet and report back.
-
@olivierlambert ALright Finally got server back up from a raid controller issue,
the Ethernet only Nic is installed, and plugged in. I see the Connection in my Switch/Network Controller But still no IP or anything in XCP MGMT, Went to Network MGMT and its still not detecting a interface
Update1 / Edit 1
I was able to get back into the Web UI for a split moment, and then something happened not sure what and it lost it again, However this time i can see the NIC in Display NIC
went to Configure MGMT interface, selected the Same ETH4 And got Config Failed " Unknown Error Occured while attempting to configure interface "
So we are making progress!
-
Have you made a
xe pif-scanafter adding the new NIC? -
@olivierlambert I did not
I ran the ctl-vsctl add xenrb0
Then add-port eth4. Pkill declient, reran dhclient it got the new IP. And thought I was good cause I was able to access XO lite interface via webUI. But about 5 mins later it yeeted the IP again.Do I need to remove the xenbr0 and eth4 port. And redo those steps to run your command? Or can I just run it?
Currently in the mist of readding all my VMs but not being able to copy and paste into the terminal is tedious with all the UUIDs
-
Okay so the right approach is NOT to tinker with the Dom0 but to use xe commands to do this. Otherwise, you might have problems like that. The right approach is to scan for the new NIC.
Our doc explains how to do so: https://docs.xcp-ng.org/networking/#add-a-new-nic
-
@olivierlambert welll dang! Alright.
So to rectify this. Do I need to delete the configs and then proceed via your link?
-
I admin I never modified the dom0 manually with OVS commands, so I would probably network reset and then pif scan.
-
@olivierlambert perfect.
Thank you tons for the support!! Ill get back home from work and give this a try and will report back!
-
@olivierlambert thank you again for all the help! Finally able to get it reconnected and able to access the web UI to a extent. But now im getting pinged with this. Even tho im using the new IP obtained via Dhclient.
And I dont have the full UI installed again. Trying to get it restored now but its being tedious
I tried clearing browser cache and accessing the UI via IP, I see what its saying and all, but not sure as to why. When I pull the SSL cert in Google it pulls up the new IP. And cert dont expires for 10 years and everything seems to be fine. I guess maybe there is a old IP still lingering somewhere in XCP?
-
For some reasons, it's having "localhost" somewhere, but IDK how
Maybe try to check if there's any record of "localhost" in XAPI, you can check the network object, host, PIFs and pool (xe <object>-param-list uuid=<object UUID)
-
Very weird that you have https://localhost/ , local host only works if your browser is on the same hardware but yours is a mobile phone.
