XCP-ng 8.3 betas and RCs feedback 🚀

archw

Yesterday, as I was about to walk out of the office for a deposition, someone walked in and said the connection to oen of the VM's was dead.

I opened up Idrac to the Dell host (Dell Inc. PowerEdge R540) and found a black screen unlike any I've seen before with XCP-NG; my vague recollection was a standard linux screen with "system" or something like that. I had twenty minutes to get to the deposition so I didn't have time to do normal debugging so I rebooted the host and watched as it did a normal reboot. It came back and all was well.

Now that the dust has cleared, this is my first chance to look into what happened. Where do I start? /var/log/xensource.log? /var/log/kern.log? Something else?

Thanks!

stormi

@archw Some information at https://docs.xcp-ng.org/troubleshooting/log-files/

flakpyro

I have been installing 8.3 beta 2 on a variety of different server grade hardware in the last week. (HP DL325, HP DL20, Lenovo SR250V2) and all have worked without issues however the issue posted by myself and @rmaclachlan above in regards to networking bonds not reporting the proper speed still remains.

I am also seeing lots of xcp-networkd errors in xensource.log

Feb 23 11:41:17 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/device/vendor, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/device/vendor")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A
Feb 23 11:41:22 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/carrier, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/carrier")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A
Feb 23 11:41:22 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/device/device, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/device/device")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A
Feb 23 11:41:22 xcpng-test-01 xcp-networkd: [error||3 ||network_utils] Error in read one line of file: /sys/class/net/bond0/device/vendor, exception Unix.Unix_error(Unix.ENOENT, "open", "/sys/class/net/bond0/device/vendor")\x0ARaised by primitive operation at Xapi_stdext_unix__Unixext.with_file in file "lib/xapi-stdext-unix/unixext.ml", line 90, characters 11-40\x0ACalled from Xapi_stdext_unix__Unixext.buffer_of_file in file "lib/xapi-stdext-unix/unixext.ml" (inlined), line 177, characters 31-83\x0ACalled from Xapi_stdext_unix__Unixext.string_of_file in file "lib/xapi-stdext-unix/unixext.ml", line 179, characters 47-73\x0ACalled from Network_utils.Sysfs.read_one_line in file "ocaml/networkd/lib/network_utils.ml", line 156, characters 6-33\x0A

My bond interfaces in XO report as running at 0 b/s as well

gb.123

I have recently installed 8.3 beta and all update patches over it... It seems to be running fine for me (on AMD).. Should I go ahead and install XOSTOR over it and see if that works ?

olivierlambert

XOSTOR isn't entirely ready on 8.3 yet (at least it's not on the latest bug fixes level)

Anonabhar

So I got a weird one. I have installed the latest XCP 8.3 updates and rebooted my server. All the VM's I had on my test server worked perfectly fine except for a single Debian 9 VM that would start booting and then "power off" just as the kernel started to spit stuff on the display.

I banged around with it for a while and what I found is its a kernel crash somewhere when SMP is initialized. If I only give a single vProc to the VM, it boots normally and all works fine.

At this point, its not causing me any more problems because I jsut rebuilt the VM on something more modern (Rocky 9)

I have perserved this VM if the dev's would like to get more debugging information or wish to try anything. I can also capture the logs if its of interest.

stormi

@flakpyro If the pull request mentioned by @psafont earlier does fix it, we'll know next time we rebase on the fixed code.

stormi

@Anonabhar I don't think we have enough work bandwidth to investigate this, but it could be good, if you have enough space, to keep it for some time. If someone has a similar issue in the future, maybe this will prove useful.

archw

I noteced new updates for 8.3 this morning. Oen of teh machines will not update. When you try to run it from XOA you get this error:

pool.installPatches
{
  "hosts": [
    "7ce4f772-4391-4982-a1f9-d1de86be92cb"
  ]
}
{
  "code": "-1",
  "params": [
    "Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1",
    "",
    "Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/updater.py\", line 96, in decorator
    return func(*args, **kwargs)
  File \"/etc/xapi.d/plugins/updater.py\", line 182, in update
    return install_helper(session, args, 'update')
  File \"/etc/xapi.d/plugins/updater.py\", line 153, in install_helper
    raise error
CalledProcessError: Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1
"
  ],
  "call": {
    "method": "host.call_plugin",
    "params": [
      "OpaqueRef:fab0b7b0-de37-a996-1760-92a38cf136c2",
      "updater.py",
      "update",
      {}
    ]
  },
  "message": "-1(Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1, , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/updater.py\", line 96, in decorator
    return func(*args, **kwargs)
  File \"/etc/xapi.d/plugins/updater.py\", line 182, in update
    return install_helper(session, args, 'update')
  File \"/etc/xapi.d/plugins/updater.py\", line 153, in install_helper
    raise error
CalledProcessError: Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1
)",
  "name": "XapiError",
  "stack": "XapiError: -1(Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1, , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 119, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/updater.py\", line 96, in decorator
    return func(*args, **kwargs)
  File \"/etc/xapi.d/plugins/updater.py\", line 182, in update
    return install_helper(session, args, 'update')
  File \"/etc/xapi.d/plugins/updater.py\", line 153, in install_helper
    raise error
CalledProcessError: Command '['yum', 'update', '--disablerepo=*', '--enablerepo=xcp-ng-base,xcp-ng-updates', '-y']' returned non-zero exit status 1
)
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202403291838/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202403291838/packages/xen-api/transports/json-rpc.mjs:38:21"


If you go to the command line and do a "yum update" you get this:


Transaction Summary
===================================================================================================================================================================================
Install              ( 1 Dependent package)
Upgrade  21 Packages

Total size: 84 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  file /usr/lib64/python2.7/site-packages/xen/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/__init__.py from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/__init__.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/__init__.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/__init__.pyo from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/__init__.pyo from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python2.7/site-packages/xen/lowlevel/xc.so from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/__pycache__/__init__.cpython-36.opt-1.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/__pycache__/__init__.cpython-36.opt-1.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/__pycache__/__init__.cpython-36.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/__pycache__/__init__.cpython-36.pyc from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64
  file /usr/lib64/python3.6/site-packages/xen/lowlevel/xc.cpython-36m-x86_64-linux-gnu.so from install of xen-installer-files-4.13.5-10.42.3.xcpng8.3.x86_64 conflicts with file from package xen-dom0-tools-4.17.3-2.0.xen417.1.xcpng8.3.x86_64

Any ideas?

olivierlambert

Have you, at any time, installed something outside the XCP-ng repos?

archw

@olivierlambert

Hmmmm...while I'm gonna say a guarded "I don't think so", that is a junk machine that I use to test things so that may be a possibility.

How do I tell?

Danp

@archw Post the output of yum repolist --verbose

FWIW, read here for the proper way to install additional packages.

flakpyro

@archw I see references to xen 4.17 in your logs. Are you currently using the 4.17 test repos or was 4.17 pushed out to the main 8.3 repos?

redakula

@flakpyro

Probably - i get the same error and i am on the 4.17 testing version

stormi

@archw I think you need to enable the xcp-ng-lab repo for updating because you installed the test packages with Xen 4.17.

yum update --enablerepo=xcp-ng-lab

stormi

Note: as soon as possible, we'll switch to Xen 4.17 for everyone so you won't have to think about the fact you were using packages from xcp-ng-lab.

archw

@Danp

I think all of this is my fault and I wouldn't worry about it. After I left the last post, I went in and saw a but of stuff from the @xcp-ng-lab repo. I quasi fixed it doing a bunch of yum downgrades until I no longer got an error and and I think I fixed it. I had to remove a few things but I took a "yum list installed" from a good system and made everything the same. Again, its a dummy system that I don't really use except to test things.

Here is the output from "yum repolist --verbose"

Loading "fastestmirror" plugin
Config time: 0.010
Yum version: 3.4.3
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Setting up Package Sacks
pkgsack time: 0.004
Repo-id      : xcp-ng-base
Repo-name    : XCP-ng Base Repository
Repo-revision: 1711706182
Repo-updated : Fri Mar 29 05:56:22 2024
Repo-pkgs    : 3,599
Repo-size    : 12 G
Repo-baseurl : http://mirrors.xcp-ng.org/8/8.3/base/x86_64/,
             : http://updates.xcp-ng.org/8/8.3/base/x86_64/
Repo-expire  : 21,600 second(s) (last: Sat Mar 30 15:37:49 2024)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/xcp-ng.repo

Repo-id      : xcp-ng-updates
Repo-name    : XCP-ng Updates Repository
Repo-revision: 1693236966
Repo-updated : Mon Aug 28 11:36:06 2023
Repo-pkgs    : 2
Repo-size    : 4.6 k
Repo-baseurl : http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/,
             : http://updates.xcp-ng.org/8/8.3/updates/x86_64/
Repo-expire  : 21,600 second(s) (last: Sat Mar 30 15:37:50 2024)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/xcp-ng.repo

repolist: 3,601

stormi

@archw yum update --enablerepo=xcp-ng-lab didn't work for you?

gduperrey

A new version of xo-lite for XCP-ng 8.3 has been released:

Version: xo-lite-0.2.1-1.xcpng8.3

You can update it like this:

yum update xo-lite

For more information about the changes between version 0.1.3 or 0.2.0 and 0.2.1, you can consult this link: https://github.com/vatesfr/xen-orchestra/blob/master/%40xen-orchestra/lite/CHANGELOG.md

archw

@stormi I didn't even think to try that I just did it and its rebooting.