Netdata package is now available in XCP-ng

PC_123

@olivierlambert
After running the command "yum install netdata-ui" on 2 hosts. Only one started working.

I can reach the netdata page on the working server at: url:19999. I've rebooted the newer of the 2 servers and no matter what I've tried the netdata page doesn't appear.

If I try the command again I receive an error indicating that the package is already installed and there is nothing to do.

Is there an easy way to fix this or, what's the command to rollback this install?

Thanks to everyone in the community (especially @olivierlambert) for your hard work on this software.

olivierlambert

Double check your firewall on the "non working host". Also check if Netdata service is up

PC_123

@olivierlambert

Imagine that I don't know what to do. I've tried reviewing the Netdata documentation but it isn't specific to XCP-NG.

I ran the below code and received a response. Does that indicate where the issue could be? Also...what code do you suggest I run to check to see if Netdata is running? Why can't it bind to the host address?

[09:49 xcp-ng3 ~]# /usr/sbin/netdata
2020-06-16 09:49:55: netdata INFO  : MAIN : SIGNAL: Not enabling reaper
2020-06-16 09:49:55: netdata ERROR : MAIN : LISTENER: IPv4 bind() on ip '0.0.0.0' port 19999, socktype 1 failed. (errno 98, Address already in use)
2020-06-16 09:49:55: netdata ERROR : MAIN : LISTENER: Cannot bind to ip '0.0.0.0', port 19999
2020-06-16 09:49:55: netdata ERROR : MAIN : LISTENER: IPv6 bind() on ip '::' port 19999, socktype 1 failed. (errno 98, Address already in use)
2020-06-16 09:49:55: netdata ERROR : MAIN : LISTENER: Cannot bind to ip '::', port 19999
2020-06-16 09:49:55: netdata FATAL : MAIN :LISTENER: Cannot listen on any API socket. Exiting... # : Success

2020-06-16 09:49:55: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
2020-06-16 09:49:55: netdata INFO  : MAIN : EXIT: cleaning up the database...
2020-06-16 09:49:55: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
2020-06-16 09:49:55: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...

olivierlambert

@PC_123 said in Netdata package is now available in XCP-ng:

Address already in use

It means there's already a service listening on this port. Try to stop it and see if it's still there on this port.

stormi

@PC_123 said in Netdata package is now available in XCP-ng:

Also...what code do you suggest I run to check to see if Netdata is running?

systemctl status netdata.service

PC_123

@stormi
Green is a good.

[10:06 xcp-ng3 ~]# systemctl status netdata.service
● netdata.service - Real time performance monitoring
   Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; vendor pres                                                        et: disabled)
   Active: active (running) since Tue 2020-06-16 09:48:42 EDT; 18min ago
  Process: 1588 ExecStartPre=/usr/libexec/netdata/xcpng-iptables-restore.sh (cod                                                        e=exited, status=0/SUCCESS)
  Process: 1523 ExecStartPre=/bin/chown -R netdata:netdata /var/run/netdata (cod                                                        e=exited, status=0/SUCCESS)
  Process: 1470 ExecStartPre=/bin/mkdir -p /var/run/netdata (code=exited, status                                                        =0/SUCCESS)
  Process: 1404 ExecStartPre=/bin/chown -R netdata:netdata /var/cache/netdata (c                                                        ode=exited, status=0/SUCCESS)
  Process: 1359 ExecStartPre=/bin/mkdir -p /var/cache/netdata (code=exited, stat                                                        us=0/SUCCESS)
 Main PID: 1861 (netdata)
   CGroup: /system.slice/netdata.service
           ├─1861 /usr/sbin/netdata -P /var/run/netdata/netdata.pid -D -W set...
           ├─2035 /usr/libexec/netdata/plugins.d/apps.plugin 1
           ├─2058 /usr/libexec/netdata/plugins.d/go.d.plugin 1
           ├─2063 /usr/libexec/netdata/plugins.d/xenstat.plugin 1
           └─2069 /usr/bin/python /usr/libexec/netdata/plugins.d/python.d.plu...

PC_123

@PC_123

I ran identical code on two machines. One worked and the other didn't. The machine that didn't work is not the master of the pool. Could that be the reason? Do I need to setup the centralized reporting that was mentioned at the beginning of this thread?

stormi

If you installed netdata-ui, each instance of netdata is independant and has no knowledge of the pool setup. There's no obvious reason why it works on one and not on the other. Except if you installed netdata only instead of netdata-ui.

stormi

What's the output of iptables -L on both hosts?

PC_123

@stormi
Not working host:

[10:07 xcp-ng3 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_input_chain  tcp  --  anywhere             anywhere             tcp dpt:nbd
ACCEPT     gre  --  anywhere             anywhere
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_output_chain  tcp  --  anywhere             anywhere             tcp spt:nbd

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere             icmp any
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW udp dpt:ha-cluster
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:21064
ACCEPT     udp  --  anywhere             anywhere             multiport dports hpoms-dps-lstn,netsupport
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain xapi_nbd_input_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain xapi_nbd_output_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Working Host:

[10:28 xcp-ng2 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_input_chain  tcp  --  anywhere             anywhere             tcp dpt:nbd
ACCEPT     gre  --  anywhere             anywhere
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
xapi_nbd_output_chain  tcp  --  anywhere             anywhere             tcp spt:nbd

Chain NETDATA (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:dnp-sec

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere             icmp any
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     udp  --  anywhere             anywhere             ctstate NEW udp dpt:ha-cluster
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:21064
ACCEPT     udp  --  anywhere             anywhere             multiport dports hpoms-dps-lstn,netsupport
NETDATA    all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain xapi_nbd_input_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain xapi_nbd_output_chain (1 references)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

stormi

The firewall rule for netdata is clearly missing here on the first host. I don't know why.

stormi

What's the output of: rpm -qa | grep netdata on both hosts?

PC_123

@stormi

[10:28 xcp-ng3 ~]# rpm -qa | grep netdata
netdata-1.19.0-3.xcpng8.1.x86_64
netdata-ui-1.19.0-3.xcpng8.1.x86_64

[10:29 xcp-ng2 ~]# rpm -qa | grep netdata
netdata-1.19.0-3.xcpng8.1.x86_64
netdata-ui-1.19.0-3.xcpng8.1.x86_64

PC_123

@stormi

I'm glad you were able to spot a difference. My untrained eye still doesn't see the difference.

stormi

Can you run /usr/libexec/netdata/xcpng-iptables-restore.sh on the host where it's not working and share the output?

PC_123

@stormi

That fixed it, thank you. Any idea why the firewall rule wasn't initially created?

[10:59 xcp-ng3 ~]# /usr/libexec/netdata/xcpng-iptables-restore.sh
Applying firewall rules for netdata from /etc/sysconfig/iptables_netdata

stormi

No, I don't know. The iptables rules are rebuilt each time your host starts, and the additional rules for netdata are added when the netdata service starts.

PC_123

@stormi

I just realized your icon is from Keen4. I use to play that game when I was much younger.

Thanks again for all your help!

PC_123

@PC_123
For anyone who has this issue in the future.

The command stormi provided did fix the problem on a running machine. Unfortunately it did not persist following a reboot.

The command: iptables -F

Seems to be a better long term solution.

Thanks again to @stormi for isolating the issue.

stormi

That command I provided is run each time the netdata service starts. So the proper fix is to find out why it doesn't work. Maybe it runs too soon and a dependency needs to be added to another systemd unit.

iptables -F is NOT a fix nor a "better long term solution". It simply deletes all the iptables rules, so basically you're disabling the firewall. By the way it does not survive a reboot.