Best posts made by Andrew

@gduperrey It's working for me, but my CPUs are not covered by the update. Normal operations seem normal.

@stormi Updates ran fine. Systems seem to be working.

@stormi Post update, things are working as normal on my hosts.

@stormi Manual updates done on all my 8.2.1 machines and things are working correctly so far.

@stormi I upgraded my normal pool from 8.2.0 to 8.2.1 (staging) using yum. It took some work because of the version change my pool master got unhappy with the order I did it. My mistake with the process... I ended up upgrading and rebooting all pool members and then things were good. I abused the upgrade process and things still worked out in the end. No trouble, stuck, damaged, or lost VMs (or other resources). Things are working as they should including shared SR on NFS, ISO on NFS, VxLAN, migration, replication, and S3 delta backups. I'm not testing USB/GPU/pass-thru.

@olivierlambert

Nice YouTube video by Lawrence Systems: Is XCP-NG a Good Alternative Replacement For VMware?

@olivierlambert fixTasks branch works for me too.

@Forza I agree that the OS is responsible for caching host records. The real question is why is XO doing so many lookups repeatedly. Maybe it is actually a Node problem (in addition to code issues).

In most applications once a socket is opened to a host it stays open and does not need to do another lookup until it is closed and a new connection is made. If XO or Node is stateless and opens a new connection for each block read/write (or group of blocks) then it may do a lot of lookups. The mass lookups seems to be a sign of a lot of overhead that could be reduced to improve performance.

Yes, nscd can be a host query (DNS) cache solution (for XO source and XOA) but can the code be improved to reduce overhead and improve general performance?

Here is a quick MRTG image of DNS requests. You can see when I enabled nscd that caches lookup requests (hint, sunday night):

@stormi The update has been OK for me on a bunch of standard machines (older Intel/AMD) over the last day. Normal VM operation, migration, backups, reboots, etc.

I'm having problems with cross host VxLANs, but I can't blame the update for that. I reinitialized the XO SDN plugin and it's working again. It may have something to do with changing pool masters and rebooting the hub server.

@stormi I've had it running for 24 hours on several active machines doing the usual jobs. Seems good.

@stormi It installs and runs....

The "help" does not mention the user-agent option.

@olivierlambert Install patches from from XO source (current master) and 8.2.1. It works, but I an error.

Host install:

pool.installPatches
{
  "hosts": [
    "64b9bf4a-a991-438d-973f-60179f3d0868"
  ]
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

Pool install:

pool.installPatches
{
  "pool": "98ab0129-f9d1-ba08-c1c0-aa93c38d7fec"
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:152:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

I ran the 8.2.1-test5 full installer booted by Ventoy-1.0.69 from USB on a nice ThinkPad T430 laptop to install to another USB SSD Stick. I then built XO from source on Bullseye as a VM. Devices (except wifi/BT/TPM) work fine on the old machine and performance is as expected. If you need a test "server", the old T430 can be dual or quad core and 16G memory with two SATA internal drives (supports intel VT).

I also have 8.2.1 (updated with staging) on all my other servers.

@stormi New error report: I do get a logged ERROR several times a minute in Dom0 on all host servers. There is more than enough free memory in Dom0:

xcp-rrdd-plugins.log:Feb 17 14:01:29 xcp-ng-rncjnand xcp-rrdd-gpumon: [error||0 ||xcp-rrdd-gpumon] Unexpected error (Failure "not enough memory"), sleeping for 10 seconds...

@stormi Here's more info from the xensource.log

I found this error happens when you use XO, click on a HOST and then the ADVANCED tab.

Feb  2 18:40:32 xcp4 xapi: [debug||741 HTTPS 192.168.1.131->:::80|host.get_sched_gran R:cdd533230ce9|audit] Host.get_sched_gran: host='a87516dc-1363-450d-8384-10e9e4a131b4 (xcp4)'
Feb  2 18:40:32 xcp4 xapi: [debug||741 HTTPS 192.168.1.131->:::80|host.get_sched_gran R:cdd533230ce9|helpers] about to call script: /opt/xensource/libexec/xen-cmdline
Feb  2 18:40:32 xcp4 xapi: [debug||742 HTTPS 192.168.1.131->:::80|host.call_plugin R:4f64bd0de6ba|audit] Host.call_plugin host = 'a87516dc-1363-450d-8384-10e9e4a131b4 (xcp4)'; plugin = 'hyperthre
ading.py'; fn = 'get_hyperthreading' args = [ 'hidden' ]
Feb  2 18:40:32 xcp4 xapi: [ warn||740 HTTPS 192.168.1.131->:::80|event.from D:b61f8cdc98d8|xapi_message] get_since_for_events: no in_memory_cache!
Feb  2 18:40:32 xcp4 xapi: [debug||741 HTTPS 192.168.1.131->:::80|host.get_sched_gran R:cdd533230ce9|helpers] /opt/xensource/libexec/xen-cmdline --get-xen sched-gran succeeded [ output = '\x0A' ]
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] host.call_plugin R:4f64bd0de6ba failed with exception Server_error(-1, [ 'module' object has no attribute 'run'; ; Traceback (most recent call last):\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 98, in wrapper\x0A    return func(*args, **kwds)\x0A  File "/etc/xapi.d/plugins/hyperthreading.py", line 14, in get_hyperthreading\x0A    result = run_command(['xl', 'info', 'threads_per_core'])\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 67, in run_command\x0A    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)\x0AAttributeError: 'module' object has no attribute 'run'\x0A ])
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] Raised Server_error(-1, [ 'module' object has no attribute 'run'; ; Traceback (most recent call last):\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 98, in wrapper\x0A    return func(*args, **kwds)\x0A  File "/etc/xapi.d/plugins/hyperthreading.py", line 14, in get_hyperthreading\x0A    result = run_command(['xl', 'info', 'threads_per_core'])\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 67, in run_command\x0A    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)\x0AAttributeError: 'module' object has no attribute 'run'\x0A ])
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 1/6 xapi Raised at file ocaml/xapi/rbac.ml, line 231
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 2/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 103
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 3/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 4/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 5/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 6/6 xapi Called from file lib/backtrace.ml, line 177
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace]
Feb  2 18:40:32 xcp4 xapi: [ warn||743 HTTPS 19.168.1.131->:::80|event.from D:6e6288e090db|xapi_message] get_since_for_events: no in_memory_cache!
Feb  2 18:40:32 xcp4 xapi: [ warn||744 HTTPS 192.168.1.131->:::80|event.from D:6c41ed917a6a|xapi_message] get_since_for_events: no in_memory_cache!

@stormi I installed staging updates on a few running machines and it's good so far. No errors or strange issue. Running VMs, doing replication, etc.

I did a new install from the new ISO to an external USB SSD (as a test) and it's working. The volume name of the install ISO needs to be updated to 8.2.1 (from 8.2.0).

I see you are using "http" for the repo, what about "https" for better security?

You mean xen 4.13.4, right? That's what's I installed.

I updated a few active machines and no problems so far:
HP G5 (E5450), HP G8 (E5-2680), AMD (Opt 6380)

@stormi I'm happy to support XCP and community for these drivers and add them to the repository. I do have the hardware on hand so I can test changes and work on updates. Let me know what changes you need. It would be nice to have them during the install since they may be required network drivers.

@stormi Nope.... my mistake. Now ubuntu 20.04 and Windows 2016 boot with UEFI Secure Boot enabled.

# secureboot-certs install

No arguments provided to command install, default arguments will be used:
- PK: default
- KEK: default
- db: default
- dbx: latest

Downloading https://www.microsoft.com/pkiops/certs/MicCorKEKCA2011_2011-06-24.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicCorUEFCA2011_2011-06-27.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicWinProPCA2011_2011-10-19.crt...
Downloading https://uefi.org/sites/default/files/resources/dbxupdate_x64.bin...
Successfully installed certificates to the XAPI DB for pool.

@runevn What do you mean by half? Just for a VM?

Don't forget about Dom0. If you have dual 8 core and you have VMs with 8 cores assigned then you are using more than half because of Dom0. But if you are not using all of the CPU all of the time then performance could be worse but normally will not be worse with just one CPU.

Network uses CPU, disk uses CPU, management uses CPU, etc...

What about memory? With one CPU you only get 1/2 the memory slots and sometimes some PCI slots a tied to the second CPU.

If you have lots of cores and you pull a CPU then as long as you have enough processing power and memory for the VMs and Dom0 then performance might be better because everything runs on one node in the system (NUMA).

Why remove one CPU? Yes you can save some power.... Yes, a 22 core CPU is not free and you could build a second server (with lots of other parts).... but it would be better to just leave the hardware alone if it's working.

@kulmacet First make sure the BIOS and iLO are up to date (or for a G8, the latest/last version). There are known issues with some older versions.

Check iLO and the IML to see if there are hardware errors listed. Check the BIOS settings. Defaults are NOT always the best choice. Normal fans suddenly running fast points to very high heat/usage or hardware issues. HP likes to spin the fans fast when the server has hardware issues. Using 4 cores on a 16 core machine should not cause high load. Disable HT, at least as a test. XCP/Xen does not like HT for some older CPUs. Check IPMI SEL for additional hardware issues (from ipmitool in dom0). You can also run the HP diags and other system tests to see if it catches any issues.

I have many DL360p G8 systems and they work well. The DL160 is a cheaper hardware design and known to have occasional hardware problems but XCP should work if the system is healthy.

The IML will log hardware errors that the system (dmesg) won't see.