MajorP93

MajorP93

Hello XCP-ng community!

Since Vates released the new OpenMetrics plugin for Xen Orchestra we now have an official, built-in exporter for Prometheus metrics!

I was using xen-exporter before in order to make hypervisor internal RRD database available in the form of Prometheus metrics.
I migrated to the new plugin which works just fine.

I updated the Grafana dashboard that I was using in order to be compatible with the official OpenMetrics plugin and thought "why not share it with other users"?

In case you are interested you can find my dashboard JSON here: https://gist.github.com/MajorP93/3a933a6f03b4c4e673282fb54a68474b

It is based on the xen-exporter dashboard made by MikeDombo: https://grafana.com/grafana/dashboards/16588-xen/

In case you also use Prometheus for scraping Xen Orchestra OpenMetrics plugin in combination with Grafana you can copy the JSON from my gist, import it and you are ready to go!

Hope it helps!

Might even be a good idea to include the dashboard as an example in the Xen Orchestra documentation.

Best regards

MajorP93

@ph7 As a suggestion: it would improve readibility if you paste your logs on Vates official pastebin tool: https://paste.vates.tech/ and share the links here instead of posting the logs in multiple messages.

MajorP93

@gduperrey @rzr Hello!

Sorry for the delayed response and thanks a lot for raising that topic internally.

Due to these security patches being very critical/important I disabled DMC for all VMs that showed this issue (I think it was 6 Windows VMs) and patched the pool manually.

After that everything worked perfectly fine.

I wanted to react fast due to these security vulnerabilities and did not take the time to look at logs in /var/log etc.

Interestingly for the second update you guys published one day after (the kernel TCP vulnerability patch) RPU worked without any issues.

For now everything is stable and I will leave DMC disabled for these Windows VMs for now.

I was just suprised because this exact configuration, that included DMC was working stable for quite a few months.

However I can live without DMC on these VMs and am happy that everything is running stable.

Thank you very much for reacting fast on these security vulnerabilities!
I think you guys have a great process when it comes to this.

Best regards

MajorP93

@gleh Installed those RC packages in my test environment.
Tested basic operation. Looks good so far.
(Booted, all 3 SR connected, VMs able to start, both VHD and QCOW2)
Best regards

MajorP93

@rzr Thanks! How long does it take for these patches go be available?

[17:00 xcpng01 ~]# yum update
Geladene Plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

//EDIT: Nevermind. After running yum clean metadata they are being picked up.

MajorP93

I updated my test environment and performed a few tests:

migrating VMs back and forth between VHD based NFS SR and QCOW2 based iSCSI SR --> VMs got converted between VHD and QCOW2 just fine, live migration worked
creation of rather big QCOW2 based VMs (2.5+ TB)
NBD-enabled delta backups of a mixed set of VMs (small, big, QCOW2, VHD)

All tests worked fine so far. Only thing that I noticed: When converting VHD-based VMs to QCOW2 format I was not able to storage migrate more than 2 VMs at a time. XO said something about "not enough memory". That might be related to my dom0 in test environment only having 4GB of RAM. Maybe not related to VHD to QCOW2 migration path. I never saw this error in my live environment where all node's dom0 have 8GB RAM.

Update candidate looks good so far from my point of view.

MajorP93

@MathieuRA I disabled Traefik and reverted to my old XO config (port 443, ssl encryption, http to https redirection), rebuild the docker container using your branch and tested:

it is working fine on my end now

Thank you very much!

I did not expect this to get fixed so fast!

MajorP93

@acebmxer Hello, thanks for sharing this here!

Me and my team have been waiting for this release!

I signed up for the Veeam newsletter hoping I would get notified once Veeam 13.1 (an XCP-ng plugin) gets released but they seem to be a bit slow.

So I found out about this release due to your post.

Will start testing!

MajorP93

I updated to branch "mra-fix-rest-memory-leak".
I will look at backup job results tomorrow and report back.

MajorP93

@Pilow said in backup mail report says INTERRUPTED but it's not ?:

@MajorP93 you say to have 8GB Ram on XO, but it OOMkills at 5Gb Used RAM.

did you do those additionnal steps in your XO Config ?

You can increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB).
Note that simply increasing the RAM for the VM is not enough.
You must also edit the service file (/etc/systemd/system/xo-server.service) 
to increase the memory allocated to the xo-server process itself.

You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use 3584 for the memory value below.

- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
The last step is to refresh and restart the service:

$ systemctl daemon-reload
$ systemctl restart xo-server

Interesting!
I did not know that it is recommended to set "--max-old-space-size=" as a startup parameter for Node JS with the result of (total system ram - 512MB).
I added that, restarted XO and my backup job.

I will test if that gives my backup jobs more stability.
Thank you very much for taking the time and recommending the parameter.

MajorP93

@gduperrey @rzr Hello!

Sorry for the delayed response and thanks a lot for raising that topic internally.

Due to these security patches being very critical/important I disabled DMC for all VMs that showed this issue (I think it was 6 Windows VMs) and patched the pool manually.

After that everything worked perfectly fine.

I wanted to react fast due to these security vulnerabilities and did not take the time to look at logs in /var/log etc.

Interestingly for the second update you guys published one day after (the kernel TCP vulnerability patch) RPU worked without any issues.

For now everything is stable and I will leave DMC disabled for these Windows VMs for now.

I was just suprised because this exact configuration, that included DMC was working stable for quite a few months.

However I can live without DMC on these VMs and am happy that everything is running stable.

Thank you very much for reacting fast on these security vulnerabilities!
I think you guys have a great process when it comes to this.

Best regards

MajorP93

Aaaaand it happened for the next VM.
Apart from updating XO I did not change anything...
It looks like RPU is failing on every single VM that has DMC enabled...
To me it looks like a regression

//EDIT:
the weird thing is that it fails with the same error even with DMC disabled...
It also fails when manually trying to live migrate the VM.
This particular VM is a Windows Server VM, XCP-ng tools version: Management agent 9.2.385-0
What leaves me baffled is that all of this has been working just fine for several months...

vm.migrate
{
  "vm": "25cf2537-c8f9-a3ef-ca68-6621351544c1",
  "migrationNetwork": "1ee81d68-6614-f134-f4b4-6bb5399bf66a",
  "targetHost": "d984660e-c9a9-4981-aac8-de3df67833a9"
}
{
  "code": "BALLOONING_TIMEOUT_BEFORE_MIGRATION",
  "params": [
    "OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813"
  ],
  "task": {
    "uuid": "dd29dc7b-62a6-1820-79d7-f447b7a5dcf6",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20261103T16:17:54Z",
    "finished": "20261103T16:20:44Z",
    "status": "failure",
    "resident_on": "OpaqueRef:73f0c67e-d88c-6ed0-9070-ca96f899319f",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "BALLOONING_TIMEOUT_BEFORE_MIGRATION",
      "OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2802))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2859))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2122))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2121))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 239))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 244))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2257))((process xenopsd-xc)(filename result.ml)(line 23))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2250))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2853))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2589))((process xenopsd-xc)(filename list.ml)(line 121))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2582))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2973))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 439))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2986))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 439))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2885))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3432))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3442))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3463))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 192))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3483))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3654))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 355))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 361))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 386))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 527))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3492))((process xapi)(filename lib/backtrace.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 551))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2605))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "BALLOONING_TIMEOUT_BEFORE_MIGRATION(OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813)",
  "name": "XapiError",
  "stack": "XapiError: BALLOONING_TIMEOUT_BEFORE_MIGRATION(OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813)
    at Function.wrap (file:///etc/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///etc/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1374:24)
    at file:///etc/xen-orchestra/packages/xen-api/index.mjs:1408:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1398:12)
    at Xapi._watchEvents (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1605:14)"
}

MajorP93

Hello,

thanks for publishing these updates!

However did you guys change anything regarding rolling pool update?

I am using latest XO from sources.

It's the third time I am getting this:

I am using dynamic memory control.

During last few rounds of updates (last few months) rolling pool update was working just fine.
This is the third VM today for which RPU is complaining about ballooning / DMC.

For the first two VMs I disabled DMC and re-tried RPU.
It progressed to the next VM but then failed again with the error shown on the screenshot.

In the past RPU was working fine with DMC hence the question regarding RPU changes?

Thanks and best regards

MajorP93

Last time I tested CBT enabled backups the backup job would sometimes create a full backup for a given VM instead of an incremental one.
This bug got triggered by live migrations. When a VM got live migrated it got processed as full backup by backup jobs even after "normal" live migrations (no storage live migration).

MajorP93

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

MajorP93

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

MajorP93

@acebmxer Very interesting. Thanks for sharing! I will wait until Veeam issued a new (hotfix) release before evaluating Veeam as a replacement for Xen Orchestra Backup.

MajorP93

@gleh Installed those RC packages in my test environment.
Tested basic operation. Looks good so far.
(Booted, all 3 SR connected, VMs able to start, both VHD and QCOW2)
Best regards

MajorP93

@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

MajorP93

Hello, so I got access to the test environment back and was able to install this set of patches.

I actually installed them on the XCP-ng host before you guys released them to the stable repository.

I ran:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates

and rebooted the system.

Unfortunately I have to say that this is the first time that patches broke my system.

While the XCP-ng host is still able to boot, it is not longer able to mount my SRs.

I have 2 SR in this test environment:

1x Linstor vSAN iSCSI configured as QCOW2
1x TrueNAS Core NFSv3 configured as VHD

I spent some hours troubleshooting this and appearently it is caused by jumbo frames no longer working after applying these XCP-ng patches.

The hypervisor / storage network in this testing environment is using jumbo frames everywhere (all switches involved, all storage systems).
Prior to installing the updates everything was working fine.

Now I can not ping the storage systems anymore using jumbo frames.
(ping -M do -s 8972 ...)

Hence the tasks that are meant to mount the SRs are stuck forever:

[14:38 xcpng-test01 ~]# xe task-list
uuid ( RO)                : a6ba1324-bfe1-8aca-c5d1-d7cc57d37cca
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : f9c76606-e516-553f-706c-ff52bc303e2d
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

The interesting thing is that "ip a" is still showing MTU 9000:

[14:44 xcpng-test01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:1c:22:40:4a:d2 brd ff:ff:ff:ff:ff:ff
8: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
9: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
10: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
11: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
12: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
13: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
    inet 10.10.160.24/24 brd 10.10.160.255 scope global xapi1
       valid_lft forever preferred_lft forever

Can't start XO VM right now as it lives on one of the SR but XO Lite is also still showing jumbo frames being enabled:

Was something changed in this set of patches that could cause this issue?

Thanks and best regards