Netdata package is now available in XCP-ng
-
Then check if it's streaming correctly to the target
-
@olivierlambert
BTW Will Changing Hostname on the server break this?
I just change it's hostname recently -
Yes it's probably that.
-
@olivierlambert
Well Anyway to fix it or reinstall it? -
You should take a look in the netdata config on your host and check if it's streaming to the right XOA, check Netdata logs and so on.
-
@olivierlambert Any idea where's the netdata config file location?
-
/etc/netdata/streaming.conf
IIRC. -
# do not edit, managed by XCP-ng [stream] # Enable this on slaves, to have them send metrics. enabled = yes destination = tcp:*********.****.****:19999 api key = 0b607150-79c6-11eb-9575-c210359af93e timeout seconds = 60 default port = 19999 send charts matching = * buffer size bytes = 1048576 reconnect delay seconds = 5 initial clock resync iterations = 60
I found these in stream.conf
-
Is the destination matches XO IP address?
-
@olivierlambert
Yup It does match XOA's IP Address -
So check netdata logs in
/var/log
to see if there's any error. -
This post is deleted! -
I see this, which looks good at first sight:
2021-05-05 14:53:06: netdata INFO : STREAM_SENDER[localhost] : STREAM localhost [send to tcp:XXX:19999]: connecting... 2021-05-05 14:53:06: netdata INFO : STREAM_SENDER[localhost] : STREAM localhost [send to tcp:XXX:19999]: initializing communication... 2021-05-05 14:53:06: netdata INFO : STREAM_SENDER[localhost] : STREAM localhost [send to tcp:XXX:19999]: waiting response from remote netdata... 2021-05-05 14:53:06: netdata INFO : STREAM_SENDER[localhost] : STREAM localhost [send to tcp:XXX:19999]: established communication - ready to send metrics... 2021-05-05 14:53:07: netdata INFO : PLUGIN[tc] : STREAM localhost [send]: sending metrics..
(caution: your log contains the IP address you masked earlier)
-
@stormi Oh No, have any idea what might be the problem?
-
No, I don't know where the problem resides exactly.
-
Hello!
We have recently added some ml capabilities to netdata from netdata agent v1.32.0 onwards. Basically, its fairly lightweight unsupervised anomaly detection, so in addition to collecting raw metrics each second for about 1% cpu of one core (typically) and no extra storage the agent also can produce an "anomaly bit" that's 1 if recent data "looks" anomalous or 0 for normal.
Some of our users have asked about if this ML capabilities would work on netdata within XCP-ng.
I'm trying to figure out if/how i might go about answering that. I work on the ML part of all this so am a bit (very) naive on the packaging side of things.
I'm wondering if anyone would be able to help me try figure out if the ML features of the netdata agent would be available via XCP-ng?
-
Hello @andrewm4894
Sure, what do you need to know? Let me add @stormi in the conversation
-
@andrewm4894 I am really looking forward to seeing how this update goes because using Netdata inside of XCP-NG is something that I want to do an updated video on to discuss performance and tesing.
-
@olivierlambert basically that if I was using XCP-ng and I set
enabled = yes
in the[ml]
section of thenetdata.conf
if it would just work as normal.Once a user makes that config change they would get the new "Anomaly Detection" menu
What this means then is that within each chart there will also be the new
anomaly-bit
corresponding to each raw metric value.For example if you add
options=anomaly-bit
then you get the anomaly bit instead of the raw value, for example (will probably be all 0 since all normal on that server) https://london.my-netdata.io/api/v1/data?chart=system.cpu&options=anomaly-bitFor users who then claim their nodes into netdata cloud this powers the anomaly advisor feature.
Here is a quick example of me doing a k6 load test against our demo servers.
So i guess (apologies for rambling a bit) my questions are:
- What netdata agent version is available in XCP-ng by default?
- Do you know if there are any custom build flags passed to it as part of the packaging for XCP-ng? e.g sometimes package maintainers might pass things like
--disable-cloud
or--disable-ml
for various reasons so im just trying to double check if anything there that might preclude things.
I think its mainly down to the netdata version that's included by default and if any custom flags passed as part of that packaging (if that makes sense).
-
The netdata version is rather old because XCP-ng 8.2 is a LTS and netdata sadly doesn't have long term maintenance branches that would guarantee an absence of regressions. And I've had serious issues with netdata on XCP-ng in the past (disk full due to a bug that I reported at the time, fixed since but now I completely disable disk writes and let only netdata use a limited RAM database, just in case), so I'm not eager to update often as long as it is working and no unpatchable major security issues are found.
And given the pace at which netdata evolves, each update is a significant amount of packaging work for me
I will provide an update at some point, in the testing repository at least, but I don't know when exactly.