Unable to Access MGMT interface/ No NICS detected

CloudX0520

So TLDR other day my server shut off from storms, and it refused to come back up due to no NIC's / IP detected. Did a whole bunch of commands, network reset, xcp network reset, and still no dice.

Detailed version.
R720XD everything has been running fine for months until the other day a storm came in and knocked everything off line ( i guess my 1500va 1000W UPS isnt enough for a R720XD nearly idle lol )
99% of everything came back up no problems minus this server, went to see it said no MGMT interface/IP detected. o okay cool no problem ill just go in and give it the ip again and boom money right? nope,

This server on the Switch/UCG Fiber side shows a IP of 10.0.100.26, and using IP a I see the same thing. Cool we are solid, Nope, Still no dice.

So then me and GPT started to go ham on running commands, reseting the network, checking XAPI status, the workssss And still no dice.

When Running OVS-VSCTL SHOW
I see the bridge Xenbr0, i see the port, the interface, the type, and then i also see eth3 interface.
And then i went inside the network-scripts and checked the ifcfg-eth3 and ifcfg-xenbr0 and added those in since it was empty, saved and exited, ran SYSTEMCTL RESTART NETWORK and SYSTEMCTL RESTART XAPI, AND STATUS, and came back active and running Ran XE PIF-list Error

So i ran Through did these commands
xe-toolstack-restart
systemctl restart xapi
Then pinged 8.8.8.8 and it did fine, so i thought i was good

but hit a error on xe host-list

then i ran
ps aux | grep xapi

Then this is where i gave up, because everything i tried with GPT and my limited knowledge yielded no results.

I have no issue Reinstalling XCP-NG buttttt 1 VM has some crucial data that i need to get pulled off before i wipe and Reinstall.

Im currently looking for a secondary NIC to see if maybeeee my onboard NIC is dying out.

IIRC Im on Version 8.3

Is there a command or something i can run that validates and repairs files or anything like that?

olivierlambert

You need to take a look at XAPI log to check why XAPI can't start. It might help to pinpoint the problem. Also doing a dmesg to watch about anything suspicious is a good practice.

CloudX0520

@olivierlambert is there anything in particular I need to keep a eye /look for? Or maybe pull the logs in smaller sections since im having to Idrac into the machine. It wont let me scroll the logs

olivierlambert

Anything with "error" might be interesting, kernel oops or whatever network driver problem.

CloudX0520

@olivierlambert Ran dmesg and its showing a WHOLEEEE bunch but i cant see up past about 40 lines up, and either Idrac, Or the Shell wont let me scroll up.

Running This command yields me a metric tonnnn of information ( last 100 Lines )
tail -f /var/log/xensource.log

and its not letting me scroll up or down to see much more what the screen shows sadly.

olivierlambert

Gut feeling while watching your dmesg log: your Broadcom NIC is in "hybrid mode" between FC and ethernet. That might trigger your problem somehow. I would flash it to get in ethernet only. Check the exact NIC you have to see how to do this.

CloudX0520

@olivierlambert I'll give that a shot and see! The NIC ive been using has 2 sfp ports and 2 ethernet ports. ( on board ) and I found a new nic laying around just ethernets I dropped in last night. That imma see what happens when I get home.

If it dont work ill flash the on board sfp/ethernet and report back.