Hello!

We have recently added some ml capabilities to netdata from netdata agent v1.32.0 onwards. Basically, its fairly lightweight unsupervised anomaly detection, so in addition to collecting raw metrics each second for about 1% cpu of one core (typically) and no extra storage the agent also can produce an "anomaly bit" that's 1 if recent data "looks" anomalous or 0 for normal.

Some of our users have asked about if this ML capabilities would work on netdata within XCP-ng.

I'm trying to figure out if/how i might go about answering that. I work on the ML part of all this so am a bit (very) naive on the packaging side of things.

I'm wondering if anyone would be able to help me try figure out if the ML features of the netdata agent would be available via XCP-ng?

We are actually planning v1.35 stable release for next week so maybe, if not too much hassle (but I can imagine it's certainly non trivial, so defo is some hassle ), we could try see about updating to that on your end with understanding that's like a good base version for the ml stuff and then can leave it at that for a while maybe if ends up being not too painful on your end.

Totally up to you guys, but I can try help any way I can.

Qq, what would be the best way for me to try spin up a sort of test or dev XCP-ng env for me to try things out on? Or is there sort of hardware involved such that this might not be so easy. In my mind I'm imagining spinning up a VM lol which probably shows my level of naivety

@olivierlambert yep - OpenMetrics is a big deal and totally a standard we want to handle.

https://www.netdata.cloud/blog/release-1-24/

Mainly via the Prometheus collector at the moment but iirc there is still one or two OpenMetrics types we need to do some work to cover - eg Histograms is one we working on at the moment.

@olivierlambert @stormi i see i see. In terms of the ML stuff it's fairly stable (admittedly i'm being subjective here - we have been dogfooding it for the last few months and had it under beta launch since March) and is disabled by default. At some stage we will have it enabled by default but that's much further down the road.

Of course it is still evolving at the same time - for example here is a big discussion about how we can extend the training window beyond the default 4 hours etc which obviously will involve more work and changes.

So in terms of where we are now the base ML functionality that's there I consider fairly stable and would be enough to be a useful feature some users might like.

But I totally get that it's a lot of work for you guys too and indeed we are building a lot of new features, especially around ML and related features and so the agent is for sure still evolving a lot.

I'm not really sure where that leaves us. I will tag in some of our SRE folks to see if anything we can do to help ease some pain at all in some way.

I think it's a netinstall like this that maybe I need to start with

https://xcp-ng.org/docs/install.html#iso-installation

And then I "have" the iso on the VM and so can go as normal from there.

If so, Amy tips on what gcp VM if any might be easiest to try start with?

Cheers!

@olivierlambert cool, going to give it a go. Do you know of any GCP or AWS based guides or material to help?

I don't actually have any physical hardware available so all the use and boot stuff kinda throws me off (maybe I'm over thinking it). Maybe there is a way to boot a gcp VM or any images I could find in some gcp marketplace or anything?

Going to do more googling and reading but just mentioning in case you know of any specific stuff. I see @lawrencesystems also has some cool yt videos which have been really useful in learning about the concepts and ideas, but I'm kinda looking for a some gcp or aws based tutorial or steps.

We are actually planning v1.35 stable release for next week so maybe, if not too much hassle (but I can imagine it's certainly non trivial, so defo is some hassle ), we could try see about updating to that on your end with understanding that's like a good base version for the ml stuff and then can leave it at that for a while maybe if ends up being not too painful on your end.

Totally up to you guys, but I can try help any way I can.

Qq, what would be the best way for me to try spin up a sort of test or dev XCP-ng env for me to try things out on? Or is there sort of hardware involved such that this might not be so easy. In my mind I'm imagining spinning up a VM lol which probably shows my level of naivety

@olivierlambert yep - OpenMetrics is a big deal and totally a standard we want to handle.

https://www.netdata.cloud/blog/release-1-24/

Mainly via the Prometheus collector at the moment but iirc there is still one or two OpenMetrics types we need to do some work to cover - eg Histograms is one we working on at the moment.

@olivierlambert @stormi i see i see. In terms of the ML stuff it's fairly stable (admittedly i'm being subjective here - we have been dogfooding it for the last few months and had it under beta launch since March) and is disabled by default. At some stage we will have it enabled by default but that's much further down the road.

Of course it is still evolving at the same time - for example here is a big discussion about how we can extend the training window beyond the default 4 hours etc which obviously will involve more work and changes.

So in terms of where we are now the base ML functionality that's there I consider fairly stable and would be enough to be a useful feature some users might like.

But I totally get that it's a lot of work for you guys too and indeed we are building a lot of new features, especially around ML and related features and so the agent is for sure still evolving a lot.

I'm not really sure where that leaves us. I will tag in some of our SRE folks to see if anything we can do to help ease some pain at all in some way.

@olivierlambert basically that if I was using XCP-ng and I set enabled = yes in the [ml] section of the netdata.conf if it would just work as normal.

Once a user makes that config change they would get the new "Anomaly Detection" menu

What this means then is that within each chart there will also be the new anomaly-bit corresponding to each raw metric value.

For example if you add options=anomaly-bit then you get the anomaly bit instead of the raw value, for example (will probably be all 0 since all normal on that server) https://london.my-netdata.io/api/v1/data?chart=system.cpu&options=anomaly-bit

For users who then claim their nodes into netdata cloud this powers the anomaly advisor feature.

Here is a quick example of me doing a k6 load test against our demo servers.

So i guess (apologies for rambling a bit) my questions are:

1. What netdata agent version is available in XCP-ng by default?
2. Do you know if there are any custom build flags passed to it as part of the packaging for XCP-ng? e.g sometimes package maintainers might pass things like --disable-cloud or --disable-ml for various reasons so im just trying to double check if anything there that might preclude things.

I think its mainly down to the netdata version that's included by default and if any custom flags passed as part of that packaging (if that makes sense).

Hello!

We have recently added some ml capabilities to netdata from netdata agent v1.32.0 onwards. Basically, its fairly lightweight unsupervised anomaly detection, so in addition to collecting raw metrics each second for about 1% cpu of one core (typically) and no extra storage the agent also can produce an "anomaly bit" that's 1 if recent data "looks" anomalous or 0 for normal.

Some of our users have asked about if this ML capabilities would work on netdata within XCP-ng.

I'm trying to figure out if/how i might go about answering that. I work on the ML part of all this so am a bit (very) naive on the packaging side of things.

I'm wondering if anyone would be able to help me try figure out if the ML features of the netdata agent would be available via XCP-ng?