XCP-ng 8.3 betas and RCs feedback 🚀

stormi

@xerxist The Linux kernel is not exactly the heart of XCP-ng. Xen is. Also, the threat model is different from that of a Linux distribution, because the main threat here comes from VMs (privilege escalation, information disclosure, DoS...), and this is taken very deep care of, at every level.

XCP-ng's management network being meant to be on a dedicated network, not exposed to direct attackers, makes network attacks a lower threat but of course doesn't negate it so it still is to be taken into account.

Your concerns are valid, especially regarding how to make an auditor accept that it is actually maintained for the scope of XCP-ng's needs, and we're looking how to document it.

archw

Yesterday, as I was about to walk out of the office for a deposition, someone walked in and said the connection to oen of the VM's was dead.

I opened up Idrac to the Dell host (Dell Inc. PowerEdge R540) and found a black screen unlike any I've seen before with XCP-NG; my vague recollection was a standard linux screen with "system" or something like that. I had twenty minutes to get to the deposition so I didn't have time to do normal debugging so I rebooted the host and watched as it did a normal reboot. It came back and all was well.

Now that the dust has cleared, this is my first chance to look into what happened. Where do I start? /var/log/xensource.log? /var/log/kern.log? Something else?

Thanks!

stormi

@archw Some information at https://docs.xcp-ng.org/troubleshooting/log-files/

flakpyro

I have been installing 8.3 beta 2 on a variety of different server grade hardware in the last week. (HP DL325, HP DL20, Lenovo SR250V2) and all have worked without issues however the issue posted by myself and @rmaclachlan above in regards to networking bonds not reporting the proper speed still remains.

I am also seeing lots of xcp-networkd errors in xensource.log

Feb 23 11:41:17 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/device/vendor, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/device/vendor")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A
Feb 23 11:41:22 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/carrier, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/carrier")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A
Feb 23 11:41:22 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/device/device, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/device/device")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A
Feb 23 11:41:22 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/device/vendor, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/device/vendor")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A

My bond interfaces in XO report as running at 0 b/s as well

gb.123

I have recently installed 8.3 beta and all update patches over it... It seems to be running fine for me (on AMD).. Should I go ahead and install XOSTOR over it and see if that works ?

olivierlambert

XOSTOR isn't entirely ready on 8.3 yet (at least it's not on the latest bug fixes level)

Anonabhar

So I got a weird one. I have installed the latest XCP 8.3 updates and rebooted my server. All the VM's I had on my test server worked perfectly fine except for a single Debian 9 VM that would start booting and then "power off" just as the kernel started to spit stuff on the display.

I banged around with it for a while and what I found is its a kernel crash somewhere when SMP is initialized. If I only give a single vProc to the VM, it boots normally and all works fine.

At this point, its not causing me any more problems because I jsut rebuilt the VM on something more modern (Rocky 9)

I have perserved this VM if the dev's would like to get more debugging information or wish to try anything. I can also capture the logs if its of interest.

stormi

@flakpyro If the pull request mentioned by @psafont earlier does fix it, we'll know next time we rebase on the fixed code.

stormi

@Anonabhar I don't think we have enough work bandwidth to investigate this, but it could be good, if you have enough space, to keep it for some time. If someone has a similar issue in the future, maybe this will prove useful.

archw

I noteced new updates for 8.3 this morning. Oen of teh machines will not update. When you try to run it from XOA you get this error:

pool.installPatches
{
  "hosts": [
    "7ce4f772-4391-4982-a1f9-d1de86be92cb"
  ]
}
{
  "code": "-1",
  "params": [
    "Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1",
    "",
    "Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/updater.py\", line 96, in decorator
    return func(*args, **kwargs)
  File \"/etc/xapi.d/plugins/updater.py\", line 182, in update
    return install_helper(session, args, 'update')
  File \"/etc/xapi.d/plugins/updater.py\", line 153, in install_helper
    raise error
CalledProcessError: Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1
"
  ],
  "call": {
    "method": "host.call_plugin",
    "params": [
      "OpaqueRef:fab0b7b0-de37-a996-1760-92a38cf136c2",
      "updater.py",
      "update",
      {}
    ]
  },
  "message": "-1(Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1, , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/updater.py\", line 96, in decorator
    return func(*args, **kwargs)
  File \"/etc/xapi.d/plugins/updater.py\", line 182, in update
    return install_helper(session, args, 'update')
  File \"/etc/xapi.d/plugins/updater.py\", line 153, in install_helper
    raise error
CalledProcessError: Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1
)",
  "name": "XapiError",
  "stack": "XapiError: -1(Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1, , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/updater.py\", line 96, in decorator
    return func(*args, **kwargs)
  File \"/etc/xapi.d/plugins/updater.py\", line 182, in update
    return install_helper(session, args, 'update')
  File \"/etc/xapi.d/plugins/updater.py\", line 153, in install_helper
    raise error
CalledProcessError: Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1
)
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202403291838/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202403291838/packages/xen-api/transports/json-rpc.mjs:38:21"


If you go to the command line and do a "yum update" you get this:


Transaction Summary
===================================================================================================================================================================================
Install              ( 1 Dependent package)
Upgrade  21 Packages

Total size: 84 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  file /usr/lib64/python2.7/site-packages/xen/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/__init__.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/__init__.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/__init__.pyo from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/__init__.pyo from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/xc.so from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/__pycache__/__init__.cpython-36.opt-1.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/__pycache__/__init__.cpython-36.opt-1.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/__pycache__/__init__.cpython-36.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/__pycache__/__init__.cpython-36.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/xc.cpython-36m-x86_64-linux-gnu.so from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64

Any ideas?

olivierlambert

Have you, at any time, installed something outside the XCP-ng repos?

archw

@olivierlambert

Hmmmm...while I'm gonna say a guarded "I don't think so", that is a junk machine that I use to test things so that may be a possibility.

How do I tell?

Danp

@archw Post the output of yum repolist --verbose

FWIW, read here for the proper way to install additional packages.

flakpyro

@archw I see references to xen 4.17 in your logs. Are you currently using the 4.17 test repos or was 4.17 pushed out to the main 8.3 repos?

redakula

@flakpyro

Probably - i get the same error and i am on the 4.17 testing version

stormi

@archw I think you need to enable the xcp-ng-lab repo for updating because you installed the test packages with Xen 4.17.

yum update --enablerepo=xcp-ng-lab

stormi

Note: as soon as possible, we'll switch to Xen 4.17 for everyone so you won't have to think about the fact you were using packages from xcp-ng-lab.

archw

@Danp

I think all of this is my fault and I wouldn't worry about it. After I left the last post, I went in and saw a but of stuff from the @xcp-ng-lab repo. I quasi fixed it doing a bunch of yum downgrades until I no longer got an error and and I think I fixed it. I had to remove a few things but I took a "yum list installed" from a good system and made everything the same. Again, its a dummy system that I don't really use except to test things.

Here is the output from "yum repolist --verbose"

Loading "fastestmirror" plugin
Config time: 0.010
Yum version: 3.4.3
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Setting up Package Sacks
pkgsack time: 0.004
Repo-id      : xcp-ng-base
Repo-name    : XCP-ng Base Repository
Repo-revision: 1711706182
Repo-updated : Fri Mar 29 05:56:22 2024
Repo-pkgs    : 3,599
Repo-size    : 12 G
Repo-baseurl : http://mirrors.xcp-ng.org/8/8.3/base/x86_64/,
             : http://updates.xcp-ng.org/8/8.3/base/x86_64/
Repo-expire  : 21,600 second(s) (last: Sat Mar 30 15:37:49 2024)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/xcp-ng.repo

Repo-id      : xcp-ng-updates
Repo-name    : XCP-ng Updates Repository
Repo-revision: 1693236966
Repo-updated : Mon Aug 28 11:36:06 2023
Repo-pkgs    : 2
Repo-size    : 4.6 k
Repo-baseurl : http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/,
             : http://updates.xcp-ng.org/8/8.3/updates/x86_64/
Repo-expire  : 21,600 second(s) (last: Sat Mar 30 15:37:50 2024)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/xcp-ng.repo

repolist: 3,601

stormi

@archw yum update --enablerepo=xcp-ng-lab didn't work for you?

gduperrey

A new version of xo-lite for XCP-ng 8.3 has been released:

Version: xo-lite-0.2.1-1.xcpng8.3

You can update it like this:

yum update xo-lite

For more information about the changes between version 0.1.3 or 0.2.0 and 0.2.1, you can consult this link: https://github.com/vatesfr/xen-orchestra/blob/master/%40xen-orchestra/lite/CHANGELOG.md