Posts made by apz | XCP-ng and XO forum

apz

@Don-Gould-NZ I wonder if this system got cloned as intended. I see 2 devices in XO, that will be xvda and xvdb. Yet earlier you listed it was one device with two partitions. The 512M size of the 2nd device makes me wonder if the migration turned the disk partitions into devices with partition/swap signature and nothing else.

As such since the system looks for xvda's 2nd partition for root, there's no partitions so naturally it fails.

You could get SystemRescueCD, boot off that and investigate the devices.

apz

@Kajetan321 Your disk image is presented as a block device to the VM. This is done by a kernel module called blktap and its userspace part tapdisk.

You are seeing dom0's kernel messages when these block devices are attached by blktap. It's just an informational message.

apz

@olivierlambert They operate with squeaky wheel mentality, the more "I have this too" presses that report gets, the likely it'll be someone eventually notices.

apz

@olivierlambert This appears to fix it:

https://lore.kernel.org/all/171154167446.2671062.9127105384591237363.stgit@firesoul/

apz

@olivierlambert Doesn't seem to matter, I tried various installers of the 24.04 beta, they all develop the issue with 6.8 kernel which they get after the first upgrade. I first encountered this with a headless homelab virtual.

I just brought this up here as 24.04 will be out next month and after that there will most likely be a lot of people reacting to the issue. It appears to be related running as a guest under Xen. I couldn't replicate the issue with real hardware or KVM.

apz

While running a test virtual for the upcoming Ubuntu 24.04, I noticed once it went from 6.6 to 6.8 kernel it started flooding the kernel with messages like "BUG: Bad page state in process" about virtually any process that was active at the moment. After a while the guest just hangs.

This is not an Ubuntu specific bug, but it's probably going to be pretty noticeable under Ubuntu and its spinoffs once the new release comes out.

I've created a bug against the kernel package here:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2056706

After some testing it would appear to be something that gets triggered when the guest is running under Xen. For those who have an account there, could you verify this issue happens on your machine (even with just a barebones install updated to 6.8) and then hit "this affects me too" link on the bug report.

apz

Is this release technically the point where XCP-ng starts to veer off more from the XS source as to my understanding they're already keeping some parts out of public repos nowdays?

It'd be interesting to see newer Xen for example in the future and the SMAPIv3!

apz

@stormi The other (working) machine is indeed installed with test5.

Should I care about the fact that the testing repo no longer exists, so the two first commands fail?

apz

@stormi I diffed this host (originally installed as 8.2.0) and one I installed very recently (during 8.2.1 testing phase). Their differences are:

137d136
< igc-module-4.20.17-1.xcpng8.2.x86_64
274a274
> MegaCli-8.07.14-1.noarch
394d393
< r8125-module-9.003.05-1.xcpng8.2.x86_64

The 8.2.1 installation appears to have network card modules in packages, the old installation has MegaCLI installed because of its RAID controller.

xensource.log is attached, user.log didn't yield anything, neither did daemon.log

As I was running this command again and again, it succeeded once. Out of curiosity, I tried running it again and again on my other hosts that had succeeded the last time. I could introduce a random failure on all of them, maybe 1 out of 20 or more rare.

xensource.txt

apz

@stormi I have not touched xapi.conf, iptables has one additional rule to default, which allows port 4949 in. It's copy-paste of other rules in /etc/sysconfig/iptables, save changed port.

apz

@stormi

# yum reinstall guest-templates-json guest-templates-json-data-linux guest-templates-json-data-other guest-templates-json-data-windows

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Resolving Dependencies
--> Running transaction check
---> Package guest-templates-json.noarch 0:1.9.6-1.1.xcpng8.2 will be reinstalled
---> Package guest-templates-json-data-linux.noarch 0:1.9.6-1.1.xcpng8.2 will be reinstalled
---> Package guest-templates-json-data-other.noarch 0:1.9.6-1.1.xcpng8.2 will be reinstalled
---> Package guest-templates-json-data-windows.noarch 0:1.9.6-1.1.xcpng8.2 will be reinstalled
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================
 Package                                                      Arch                              Version                                       Repository                                 Size
==============================================================================================================================================================================================
Reinstalling:
 guest-templates-json                                         noarch                            1.9.6-1.1.xcpng8.2                            xcp-ng-updates                             29 k
 guest-templates-json-data-linux                              noarch                            1.9.6-1.1.xcpng8.2                            xcp-ng-updates                             17 k
 guest-templates-json-data-other                              noarch                            1.9.6-1.1.xcpng8.2                            xcp-ng-updates                             12 k
 guest-templates-json-data-windows                            noarch                            1.9.6-1.1.xcpng8.2                            xcp-ng-updates                             14 k

Transaction Summary
==============================================================================================================================================================================================
Reinstall  4 Packages

Total download size: 73 k
Installed size: 61 k
Is this ok [y/d/N]: y
Downloading packages:
(1/4): guest-templates-json-1.9.6-1.1.xcpng8.2.noarch.rpm                                                                                                              |  29 kB  00:00:00     
(2/4): guest-templates-json-data-other-1.9.6-1.1.xcpng8.2.noarch.rpm                                                                                                   |  12 kB  00:00:00     
(3/4): guest-templates-json-data-linux-1.9.6-1.1.xcpng8.2.noarch.rpm                                                                                                   |  17 kB  00:00:00     
(4/4): guest-templates-json-data-windows-1.9.6-1.1.xcpng8.2.noarch.rpm                                                                                                 |  14 kB  00:00:00     
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         110 kB/s |  73 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : guest-templates-json-1.9.6-1.1.xcpng8.2.noarch                                                                                                                             1/4 
  Installing : guest-templates-json-data-other-1.9.6-1.1.xcpng8.2.noarch                                                                                                                  2/4 
  Installing : guest-templates-json-data-windows-1.9.6-1.1.xcpng8.2.noarch                                                                                                                3/4 
  Installing : guest-templates-json-data-linux-1.9.6-1.1.xcpng8.2.noarch                                                                                                                  4/4 
Traceback (most recent call last):
  File "/bin/create-guest-templates", line 17, in <module>
    loader.insert_templates()
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
    self._insert_template(i)
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
    conn.request("PUT", "/import_metadata?" + params, tar)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 857, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe
  Verifying  : guest-templates-json-data-other-1.9.6-1.1.xcpng8.2.noarch                                                                                                                  1/4 
  Verifying  : guest-templates-json-data-windows-1.9.6-1.1.xcpng8.2.noarch                                                                                                                2/4 
  Verifying  : guest-templates-json-data-linux-1.9.6-1.1.xcpng8.2.noarch                                                                                                                  3/4 
  Verifying  : guest-templates-json-1.9.6-1.1.xcpng8.2.noarch                                                                                                                             4/4 

Installed:
  guest-templates-json.noarch 0:1.9.6-1.1.xcpng8.2                guest-templates-json-data-linux.noarch 0:1.9.6-1.1.xcpng8.2   guest-templates-json-data-other.noarch 0:1.9.6-1.1.xcpng8.2  
  guest-templates-json-data-windows.noarch 0:1.9.6-1.1.xcpng8.2  

Complete!

apz

@stormi This command did not produce any output, so I assume the packages are fine.

apz

@stormi I ran it 5 times in a row. Always after base-windows-8.json.

apz

@stormi Result:

# /usr/bin/create-guest-templates-wrapper
Load /usr/share/xapi/vm-templates/windows-server-2012-64bit.json
Load /usr/share/xapi/vm-templates/sled-12-sp4-64bit.json
Load /usr/share/xapi/vm-templates/rhel-8.json
Load /usr/share/xapi/vm-templates/rhel-7.json
Load /usr/share/xapi/vm-templates/oel-8.json
Load /usr/share/xapi/vm-templates/sle-15-64bit.json
Load /usr/share/xapi/vm-templates/debian-9.json
Load /usr/share/xapi/vm-templates/windows-8-64bit.json
Load /usr/share/xapi/vm-templates/sles-12-sp5-64bit.json
Load /usr/share/xapi/vm-templates/base-sle-hvm.json
Load /usr/share/xapi/vm-templates/windows-10-64bit.json
Load /usr/share/xapi/vm-templates/oel-7.json
Load /usr/share/xapi/vm-templates/coreos.json
Load /usr/share/xapi/vm-templates/debian-11.json
Load /usr/share/xapi/vm-templates/windows-server-2012-r2-64bit.json
Load /usr/share/xapi/vm-templates/sles-12-sp3-64bit.json
Load /usr/share/xapi/vm-templates/windows-server-2016-64bit.json
Load /usr/share/xapi/vm-templates/gooroom-2.json
Load /usr/share/xapi/vm-templates/debian-10.json
Load /usr/share/xapi/vm-templates/windows-server-2022-64bit.json
Load /usr/share/xapi/vm-templates/other-install-media.json
Load /usr/share/xapi/vm-templates/base-sle-hvm-64bit.json
Load /usr/share/xapi/vm-templates/base-kylin-7.json
Load /usr/share/xapi/vm-templates/kylin-7.json
Load /usr/share/xapi/vm-templates/debian-8.json
Load /usr/share/xapi/vm-templates/sled-12-sp3-64bit.json
Load /usr/share/xapi/vm-templates/windows-server-2019-64bit.json
Load /usr/share/xapi/vm-templates/centos-7.json
Load /usr/share/xapi/vm-templates/base-windows-uefi.json
Load /usr/share/xapi/vm-templates/sles-12-sp4-64bit.json
Load /usr/share/xapi/vm-templates/sl-7.json
Load /usr/share/xapi/vm-templates/ubuntu-20.04.json
Load /usr/share/xapi/vm-templates/windows-10-32bit.json
Load /usr/share/xapi/vm-templates/ubuntu-16.04.json
Load /usr/share/xapi/vm-templates/rocky-8.json
Load /usr/share/xapi/vm-templates/windows-8-32bit.json
Load /usr/share/xapi/vm-templates/base-hvmlinux.json
Load /usr/share/xapi/vm-templates/almalinux-8.json
Load /usr/share/xapi/vm-templates/base-el-7.json
Load /usr/share/xapi/vm-templates/centos-8.json
Load /usr/share/xapi/vm-templates/base-windows.json
Load /usr/share/xapi/vm-templates/ubuntu-18.04.json
Load /usr/share/xapi/vm-templates/base-windows-8.json
Destroy 1c33af1c-e919-418c-ad45-85d7d6fb604a
Insert 1c33af1c-e919-418c-ad45-85d7d6fb604a
Traceback (most recent call last):
  File "/usr/bin/create-guest-templates", line 17, in <module>
    loader.insert_templates()
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
    self._insert_template(i)
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
    conn.request("PUT", "/import_metadata?" + params, tar)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 857, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe

apz

@stormi The affected host has only 2 templates, 2022 Windows and Suse 12.

apz

I upgraded 3 of my homelab hosts, all were up-to-date 8.2's before this update. One of them blurted out this right at the end of the upgrade, but I did not observe any negative consequences yet.

  Cleanup    : wsproxy-1.12.0-2.xcpng8.2.x86_64                                                                                                                                       162/162 
Traceback (most recent call last):
  File "/bin/create-guest-templates", line 17, in <module>
    loader.insert_templates()
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
    self._insert_template(i)
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
    conn.request("PUT", "/import_metadata?" + params, tar)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 857, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe

apz

@stormi I installed a fresh host from 8.2.1 Test 5 full CD image in my homelab. This host has software RAID1 which was problematic with the last installer, now it went without issues.

Now that I've installed a newer version than what's in the repos, will this work correctly in the future regarding to updates?

apz

This is a fresh 8.2 system with the secure boot features added per XCP-ng documentation. The secureboot-certs install however fails:

# secureboot-certs install

No arguments provided to command install, default arguments will be used:
- PK: default
- KEK: default
- db: default
- dbx: latest

Downloading https://www.microsoft.com/pkiops/certs/MicCorKEKCA2011_2011-06-24.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicCorUEFCA2011_2011-06-27.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicWinProPCA2011_2011-10-19.crt...
Downloading https://uefi.org/sites/default/files/resources/dbxupdate_x64.bin...
error: unable to retrieve certificate from URL: https://uefi.org/sites/default/files/resources/dbxupdate_x64.bin. Error message: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>.

If the failure can't be fixed at the network configuration level, consider downloading the certificates manually and then loading one or more of them with secureboot-certs install <PK-filename>|default <KEK-filename>|default <db-filename>|default <dbx-filename>|latest. Check secureboot-certs install -h for usage details as well as a list of the download links used by secureboot-certs install.

The system's clock is correct and the uefi.org certificate seems fine to me:

* Server certificate:
*       subject: CN=uefi.org
*       start date: Oct 19 13:50:03 2021 GMT
*       expire date: Jan 17 13:50:02 2022 GMT
*       common name: uefi.org
*       issuer: CN=R3,O=Let's Encrypt,C=US

wget on the same system says the certificate is expired. A desktop's browser was fine with it and allowed me to download the file. Is there something I missed here? I did the same kind of installation couple of months back and had no issues with that.

apz

@olivierlambert I figured it was a case of work-in-progress software and I'm fine with it knowing it will be 100% local when it's ready for showtime.

apz

@olivierlambert I think this is exactly what XCP-ng needed. Sure, management and backups with XO centrally is the main point, but at times I need up access the server locally to see what's going on, especially in situations where XO is running somewhere else and the XCP-ng server in question runs the firewall for example and for some reason the firewall never came up after a reboot.

A basic access to the virtuals' consoles and the option to start/stop/reboot them alone will get far, right now I can do that the restarting part over SSH, but not view the console.

I XO Lite would be perfect with these options in addition to being completely locally stored. Right now it seems to just load the script from somewhere else, which doesn't help if the firewall virtual is down.

But yeah, good work, excellent project. I'll be following it with great interest.