Posts made by dj423

@julien-f said in % characters in cloud-init configs render as 0 in instance:

@dj423 It should be fixed in the master branch: https://github.com/vatesfr/xen-orchestra/commit/30e6d4b4f14ddbb6ee34cd456e4ca707486df2db

• {index} is now used in place of %
• % is still working when Multiple VMs is enabled
• % and \% will be replaced by % when Multiple VMs is disabled

Wow! First of all big Thank you for getting that in place so fast. I updated, and for grins provisioned an Oracle 9.4 VM in XO using the same templated config (without the escaped /%) and it worked flawless.

snippet:
{% if distro == 'rocky' or distro == 'centos' or distro == 'ol' or distro == 'almalinux' or distro == 'fedora' or distro == 'redhat' %}
{% set group = 'wheel' %}

Thanks again Julien!

@julien-f said in % characters in cloud-init configs render as 0 in instance:

@dj423 I know it's a bit cumbersome but in the meantime, you can escape the problematic character: \%.

Ok cool, I was wondering if that would work, and my testing worked perfectly. Thank you!

@julien-f said in % characters in cloud-init configs render as 0 in instance:

@dj423only useful when the advanced setting Multiple VMs is enabled, always 0 otherwise

I found that setting in advanced, and never even knew it existed lol

I am not using that feature.

Was hoping I could just disable it and see if it changes behavior.

@julien-f

Possibly, I just need the jinja lines to render with the % characters in place.

Example:

{% elif distro == 'ubuntu' or distro == 'debian' %}
{% set group = 'sudo' %}

Is there a config setting to disable the Multiple VM setting I missed?
Thanks for your help!

@dj423

After a bit more digging, I see from the XO cloud config, to the config drive is where the characters get switched - here is the output of
xo-cli cloudConfig.getAll

id: 'e7655d9f-c442-4c0f-8e16-35bf3e5e5685',
    name: 'Prod-RL9-jinja',
    template: '## template: jinja\n' +
      '#cloud-config\n' +
      'fqdn: {name}%\n' +
      "{% set u1 = 'user' %}\n" +
      "{% set u1pass = '$6$xxxxxxK0q11t/' %}\n" +

Everything looks correct, and matches the config as in XO cloud configs. All good so far.

Mounting the cloud config drive I see something different, and why it is failing - cloud init is choking on the {0 's

mount /dev/xvda /mnt
cat /mnt/user-data
root@test1:/mnt/userdata# cat user-data

## template: jinja
#cloud-config
fqdn: RL9-jinjatest0
{0 set u1 = 'user' 0}
{0 set u1pass = 'xxx/' 0}
{0 set u1key = 'xxxx xx-ed25519-ansible' 0}

All of the % characters are now 0's. See it in the hostname as well as jinja.

Anyone know what part of the process (where it reads the cloud-init userdata and packages into the config disk) would flip all %'s to 0's?? When I run tests on the original userdata config, everything renders fine when I test it with cloud init in the VM instance, so I don't think the issue is with cloud-init.

ex: cloud-init query --format="$( cat userdata.yaml )"

When I feed the same userdata config to xo-cli vm.create everything renders fine, it just doesn't have the metadata for the hostname. Really strange. I will keep digging, and try a few test configs for giggles.

Anyone else using jinja template directives for provisioning new VMs?

Im running 8.2 XO commit a5967 and Master ba375.

I started noticing that my cloud-init configs are behaving a bit different now when I provision new VM's.

In the hostname variable for example, I have used {name}% as the hostname variable to pass the VM name meta data to cloud-init.
Example:

#cloud-config
fqdn: {name}%

Now the hostname will render with a 0 on the end, ie: somehost.example.com0 rather than the vm name of somehost.example.com.

So I removed the % from the {name}% directive and now it renders the hostname correct.

However, any cloud-init configs with jinja templating in them, (using the same % character) no longer work, since the jinja variables show up in the user-data file presented to cloud-init in the nocloud datasource as {0 if distro == ... 0} when it used to pass the jinja coding to cloud-init.

As used in the cloud-init config in XOA:

{% if distro == 'rocky' or distro == 'centos' or distro == 'ol' or distro == 'almalinux' or distro == 'fedora' or distro == 'redhat' %}

renders as in user-data file as:

{0 if distro == 'rocky' or distro == 'centos' or distro == 'ol' or distro == 'almalinux' or distro == 'fedora' or distro == 'redhat' 0}

So this hoses jinja from rendering the variables.

Anyone else notice a change in how metadata is being presented to cloud-init? Possible bug, or some change I didn't notice until now? Not a deal breaker for me, since I have other configs that work fine, just odd how it changed. If I find a solution to this, I will share here.
TIA!

Re: Cloud-init success

Had a chance to "refresh" some cloud-init configs for some later distros (Debian 12, Ubuntu 24.04, Rocky Linux 9.3, etc.) so thought I would share some example configs that are used mainly for bootstrapping new VM's and or containers.

This first one is targeted at deb based distros and sets up an Incus container host. Like most configs I use this one is pretty specific pulling key files and configs from an NFS share, rsyslog target, etc., but gives an idea what can be done for detailed provisioning of instances, aside from just slapping your SSH keys on and letting ansible take over. With this I can have a fresh VM provisioned in about 5 minutes.

I always like to run a status check after init:

root@IncusTEST20:~# cloud-init status --long
status: done
boot_status_code: enabled-by-generator
last_update: Sat, 26 Jul 2024 17:29:50 +0000
detail:
DataSourceNoCloud [seed=/dev/xvdc][dsmode=net]

Then we can query the user-data to see how it rendered from the config injected into the nocloud datasource:
root@IncusTEST20:~# cloud-init query userdata

#cloud-config
    hostname: IncusTEST20
    users:
      - name: foo
        groups: sudo
        lock_passwd: false
        passwd: $6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725
        shell: /bin/bash
      - name: bar
        groups: sudo
        lock_passwd: false
        passwd: $6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1
        ssh_authorized_keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725
        shell: /bin/bash
    locale: en_US.UTF-8
    timezone: America/New_York
    resize_rootfs: true
    mounts:
      - ["192.168.99.2:/mnt/Vault/lxd", "/mnt/lxd", "nfs", "auto,nofail,noatime,nolock,intr,tcp,actimeo=1800,user,suid", "0", "0"]
      - ["192.168.0.54:/mnt/raid/backup", "/mnt/nas", "nfs", "auto,nofail,noatime,nolock,intr,tcp,actimeo=1800,user,suid", "0", "0"]
    rsyslog:
      remotes:
        log_serv: "192.168.50.35:5140"
    write_files:
      - path: /etc/init.d/incus.sh
        owner: root:root
        permissions: 0o755
        defer: true
        content: |
          #!/bin/bash
          curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc
          sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-lts-6.0.sources
          Enabled: yes
          Types: deb
          URIs: https://pkgs.zabbly.com/incus/lts-6.0
          Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
          Components: main
          Architectures: $(dpkg --print-architecture)
          Signed-By: /etc/apt/keyrings/zabbly.asc
          EOF'
          apt update -y
          dpkg --configure -a
          apt install incus incus-ui-canonical -y
          mkdir -p /root/.config
          mkdir -p /root/.config/rclone
          mount -a
          cp /mnt/lxd/.config/.encode /root/.encode
          cp /mnt/lxd/.config/rclone.conf /root/.config/rclone/rclone.conf
          chmod 600 /root/.config/rclone/rclone.conf
          # Delete self
          rm "${0}"
    runcmd:
      - mkdir /mnt/lxd
      - mkdir /mnt/nas
      - date > /etc/birth_certificate
      - [ mount /dev/cdrom /mnt ]
      - [ bash /mnt/Linux/install.sh ]
      - [ umount /dev/cdrom ]
      - [ sh, /etc/init.d/incus.sh ]
    package_update: true
    package_upgrade: true
    packages:
      - htop
      - nano
      - curl
      - wget
      - nfs-common
      - btrfs-progs
      - bridge-utils
      - build-essential
      - rclone

Here is a more universal config with jinja templating syntax that can be targeted at many distros and using an 'if distro' will configure the instance based on the distro value in metadata:

This one targets debian, ubuntu, centos, redhat and rocky linux 8 and up - notice the ##template: jinja at the top, this allows jinja to render and process:

## template: jinja
#cloud-config
{% set u1 = 'foobar' %}
{% set u1pass = '$6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1' %}
{% set u1key = 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725' %}
{% set u2 = 'ansible' %}
{% set u2pass = '$6$10023$EKz8eWTDXQO3x7.4ff0ZNJLsl9q6RB.l8pZN9nq8BzT42zzOn7O4r./ybHeVa/l0W0/FARK/2Ttg177ywAP0Z1' %}
{% set u2key = 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqTfKkUKEGxOi62A9tCWMslqF5i9xm0aMN+ZxWgHuR6 foobar-ed25519-20240725' %}
locale: en_US.UTF-8
timezone: America/New_York
runcmd:
  - mkdir /mnt/v-nas
  - mkdir /mnt/home
  - date > /etc/birth_certificate
rsyslog:
  remotes:
    log_serv: "192.168.50.35:5140"
{% if distro == 'rocky' or distro == 'centos' or distro == 'redhat' %}
{% set group = 'wheel' %}
repo_update: true
repo_upgrade: all
yum_repos:
  epel-release:
    name: Extra Packages for Enterprise Linux 9 - Release
    baseurl: http://download.fedoraproject.org/pub/epel/9/Everything/$basearch
    enabled: true
    failovermethod: priority
    gpgcheck: true
    gpgkey: http://download.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9
package_upgrade: true
packages:
  - htop
  - nano
{% elif distro == 'ubuntu' or distro == 'debian' %}
{% set group = 'sudo' %}
package_update: true
package_upgrade: true
packages:
  - htop
  - nano
  - build-essential
users:
  - name: {{ u1 }}
    groups: {{ group }}
    lock_passwd: false
    passwd: {{ u1pass }}
    ssh_authorized_keys:
      - {{ u1key }}
    shell: /bin/bash
  - name: {{ u2 }}
    groups: {{ group }}
    lock_passwd: false
    passwd: {{ u2pass }}
    ssh_authorized_keys:
      - {{ u2key }}
    shell: /bin/bash
{%- endif %}

Just a note, to consume jinja templating, you need cloud-init 22.x or higher, with the jinja package installed in your template/image.

For anyone wanting to tinker with these, (these are not my keys or password hashes, these I just threw in for demonstration purposes and for the wiki page). The password for the accounts is "foobar" and the matching private key is:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACBKk3ypFChBsToutgPbQljLJaheYvcZtGjDfmcVoB7kegAAAKADtgJ2A7YC
dgAAAAtzc2gtZWQyNTUxOQAAACBKk3ypFChBsToutgPbQljLJaheYvcZtGjDfmcVoB7keg
AAAEC1DHPxJPEU3Ywf14x7k7IMXt1nKPvwBmG6vAXsZceiVUqTfKkUKEGxOi62A9tCWMsl
qF5i9xm0aMN+ZxWgHuR6AAAAF2Zvb2Jhci1lZDI1NTE5LTIwMjQwNzI1AQIDBAUG
-----END OPENSSH PRIVATE KEY-----

Enjoy!

@runevn
A bit more from my 'troubleshooting notes' I gathered back when I was prototyping a lot of very complex configurations and worked with Chad Smith:

Test user-data rendering: cloud-init query --format "$(sudo cloud-init query userdata)"
Test jinja template rendering: cloud-init query --format="$( cat test.cfg )" # test.cfg would be the test cloud-config.cfg file to render

Test jinja template detailed: cloud-init devel render test.cfg -d
Query metadata: cloud-init query -f {{v1.distro}}

========= Debug user-data rendering =============
cloud-init devel render /var/lib/cloud/instance/user-data.txt -d

Check actual seed file from provider

cloud-init devel render /var/lib/cloud/seed/nocloud-net/user-data -d

On a system where cloud-config wasn't honored, run: sudo cloud-init query --system # or you can add --annotate which will annotate the specific lines where an error is in the #cloud-config YAML
per Chad Smith;

All cloud-init instance data /var/lib/cloud/instance/
Userdata /var/lib/cloud/instance/user-data.txt
Datasource /var/lib/cloud/instance/datasource
metadata /run/cloud-init/instance-data.json
Seed files from provider/hypervisor: /var/lib/cloud/seed/nocloud-net

For anyone doing a deep dive into troubleshooting cloud-init configuration rendering.

@runevn

Sure, if you just want to pre-test and render configs there are a few methods. Just off the top of my head:

Verify user data in container: cloud-init query userdata # This will printout the config used to configure instance so you can see how it was rendered.

To see how cloud-init rendered your config as passed from nocloud
Validate the cloud-config:

cloud-init devel schema --system --annotate

To test a config file:
Verify cloud config schema:

cloud-init devel schema --config-file test.cfg

Example: cloud-init devel schema --config-file user-data.yaml

While in the instance, you can check the files in the nocloud datasource as well, and tweak them as needed till it passes the checks:

Run the following as a guide

mount /dev/xvdb /mnt
cd /mnt

root@Deb99990:/mnt# cloud-init schema -c user-data --annotate
Valid cloud-config: user-data


root@Deb99990:/mnt# cloud-init schema -c network-config --annotate
Valid cloud-config: network-config

If you have errors or typos in your yaml, it will point them out, so you can address the error. Then once they pass the schema check, you can copy the config to your cloud-init configs in XO. Hope that helps.

Also if you just want to test the cloud-init config process on the same VM, you can "reboot" cloud-init with: cloud-init clean --logs --reboot and it will start fresh and re-apply the configs.

Then check the results with cloud-init status --long - or if it is still running (as in the case of a container) you can run: cloud-init status --wait till it finishes.

@VGerris said in Cloud-init success:

@dj423 thank you, that is really helpful!
I also found : https://cloudinit.readthedocs.io/en/latest/reference/examples.html

Yes the docs are a huge help when crafting big nuanced config scripts. The lead maintainer Chad Smith gave me some pro-tips as well and was a huge help for me when I was learning that stuff. Yes I agree, Xcp-ng rocks!!

Best of luck!

@VGerris

Sure, I have several configs in production depending on the use-case/role for the node.

Here is a sanitized version of one of the more basic configs for Ubuntu images for reference:

#cloud-config
hostname: {name}%
users:
## Add users
  - name: user1
    groups: sudo
    lock_passwd: false
    passwd: $6$xxxx.......
    ssh_authorized_keys:
      - ssh-ed25519 <my-key-ed25519>
    shell: /bin/bash
## Add ansible acct & key
  - name: ansible
    groups: sudo
    lock_passwd: false
    passwd: $6$xxxxxx736363535353.8ahdVT625S/
    ssh_authorized_keys:
      - ssh-ed25519 <somekey>
    shell: /bin/bash
package_update: true
package_upgrade: true
packages:
  - build-essential
  - htop
  - nano
  - nfs-common
  - git
locale: en_US.UTF-8
timezone: America/New_York
runcmd:
  - mkdir /mnt/v-nas
  - date > /etc/birth_certificate
  - mount /dev/cdrom /mnt
  - bash /mnt/Linux/install.sh
  - umount /dev/cdrom

Just add your pw hashes and key strings and tweak what you need for your config. This one is pretty basic so it runs through fairly quick and loads the xen-tools.

I have a very large "universal" one for Debian, Redhat, Centos, Rocky, Ubuntu with jinja templating and all the snazzy features I can sanitize and share if you like that I use for both VM's and containers.

@olivierlambert
Yes. It should be said, that XCP-ng FAR exceeds my expectations! No matter what I throw at it, it just works fantastic.

@dj423

Update: In case anyone runs across this OVA/app, It imported fine, with no issues at all. I did not end up running it since the apps fell short of my expectations, but the process of importing it worked flawlessly.

Have a great day!

@olivierlambert
Ok thanks Oliver! I will report back how it goes.

Just curious if VirtualBox OVA files are compatible with XCP-ng's import feature?

I found this OVA for Nextcloud-OnlyOffice and I thought I would give it a go.
https://www.univention.com/products/univention-app-center/onlyoffice-document-server-with-nextcloud/

Has anyone had success importing an OVA for Virtualbox? I may try it out and report back.

Thanks!

@dj423
Second occurrence of kernel BUG error:

[237621.042604] BUG: unable to handle page fault for address: ffff9bd9bc6c9840
[237621.049589] #PF: supervisor read access in kernel mode
[237621.055044] #PF: error_code(0x0000) - not-present page

I am seeing some strange errors on the host (X1), for storage devices that no longer exist on the host. e.g.:

[331863.198883] print_req_error: I/O error, dev tde, sector 74842904
[332612.663953] block tdd: sector-size: 512/512 capacity: 178257920
[332612.711329] block tde: sector-size: 512/512 capacity: 251658240
[333679.019750] print_req_error: 86 callbacks suppressed
[333679.019750] print_req_error: I/O error, dev tdd, sector 74924032

The storage devs on host:

tdc                                                                                                   254:2    0    85G  0 disk
tda                                                                                                   254:0    0   120G  0 disk
tdf                                                                                                   254:5    0    32G  0 disk

Any idea why I would see I/O errors for storage devices that are no longer on the host? I only see this on one VM on this host, all others are fine.

@dj423

After a bit more digging, tde is "xvdb" within the VM. This block device was on the iSCSI share, so I moved it over to the NFS share, since that seems to be the only difference with that device. All other drives are on the NFS share and having no issues. May be an issue with the iSCSI SR. I will create a test VM on there to troubleshoot more.

Both VM's have an xvda (OS Drive) and a xvdb with btrfs volumes on them that LXD consumes for container instances. This "tde" device was the only block device stored on the iSCSI SR, so thinking that may be where the I/O issue stems.

I set no authentication on the iSCSI share, so I can rule out authentication issues. Or I may just avoid iSCSI since it's a thick provision process. These are only test VM's so no data is at risk.

Here is the current SR list on this host:

uuid ( RO)                : 3bf92a0f-26ea-cd46-e45c-0a896f267c3f
          name-label ( RW): XCP-ng Tools
    name-description ( RW): XCP-ng Tools ISOs
                host ( RO): <shared>
                type ( RO): iso
        content-type ( RO): iso


uuid ( RO)                : e506e70d-7351-647a-bd4a-8053b14c4c1f
          name-label ( RW): Local Storage
    name-description ( RW):
                host ( RO): X2-P320
                type ( RO): ext
        content-type ( RO): user


uuid ( RO)                : df5a8a4c-50f1-799c-7523-b32064607937
          name-label ( RW): XEN-NFS
    name-description ( RW): VM Shared Storage
                host ( RO): <shared>
                type ( RO): nfs
        content-type ( RO): user


uuid ( RO)                : 93dcfee9-1d23-e585-ca78-13cc781421fe
          name-label ( RW): ISO-NAS
    name-description ( RW): ISO Share
                host ( RO): <shared>
                type ( RO): iso
        content-type ( RO): iso


uuid ( RO)                : fc21f3a9-3651-404c-0756-155660acf953
          name-label ( RW): LXD-Pool
    name-description ( RW): LXD Pools
                host ( RO): <shared>
                type ( RO): lvmoiscsi
        content-type ( RO): user


uuid ( RO)                : d3dabb64-cf3d-bb00-70aa-baf2240ac95e
          name-label ( RW): Local Storage
    name-description ( RW):
                host ( RO): X1-M920
                type ( RO): ext
        content-type ( RO): user

LXD-Pool is the suspect SR, as the only device "tde" on this SR showed I/O errors.

Let me know if my thinking is correct on that.

@olivierlambert

Some of the ugly dmesg output from host with the troubled VM:

[242392.838917] vif vif-7-0 vif7.0: Guest Rx ready
[242421.920556] block tdd: sector-size: 512/512 capacity: 251658240
[242427.246697] block tde: sector-size: 512/512 capacity: 251658240
[242428.379056] block tdf: sector-size: 512/512 capacity: 251658240
[243384.664175] block tde: sector-size: 512/512 capacity: 251658240
[243385.760070] block tdf: sector-size: 512/512 capacity: 251658240
[244310.448680] block tde: sector-size: 512/512 capacity: 251658240
[244311.556701] block tdf: sector-size: 512/512 capacity: 251658240
[244312.702292] block tdg: sector-size: 512/512 capacity: 251658240
[244321.783435] device vif8.0 entered promiscuous mode
[244336.272665] block tda: sector-size: 512/512 capacity: 178257920
[244337.035240] device vif7.0 left promiscuous mode
[244337.251070] vif vif-8-0 vif8.0: Guest Rx ready
[244362.015783] block tdc: sector-size: 512/512 capacity: 178257920
[244367.775702] block tde: sector-size: 512/512 capacity: 178257920
[244368.875340] block tdf: sector-size: 512/512 capacity: 178257920
[244984.216622] block tde: sector-size: 512/512 capacity: 178257920
[244985.375864] block tdf: sector-size: 512/512 capacity: 178257920
[245602.874372] block tde: sector-size: 512/512 capacity: 178257920
[245604.076988] block tdf: sector-size: 512/512 capacity: 178257920
[245605.204176] block tdg: sector-size: 512/512 capacity: 178257920
[245627.262457] device vif9.0 entered promiscuous mode
[245640.879281] block tda: sector-size: 512/512 capacity: 251658240
[245641.649783] device vif8.0 left promiscuous mode
[245641.898617] vif vif-9-0 vif9.0: Guest Rx ready
[246215.439267] block tdd: sector-size: 512/512 capacity: 178257920
[246215.479136] block tde: sector-size: 512/512 capacity: 251658240
[247387.052767] print_req_error: I/O error, dev tde, sector 75145216
[247387.052820] print_req_error: I/O error, dev tde, sector 75145304
[247387.052865] print_req_error: I/O error, dev tde, sector 75145392
[247387.052930] print_req_error: I/O error, dev tde, sector 75145480
[247387.053039] print_req_error: I/O error, dev tde, sector 75145568
[247387.053087] print_req_error: I/O error, dev tde, sector 75145656
[247387.053137] print_req_error: I/O error, dev tde, sector 75145744
[247387.053183] print_req_error: I/O error, dev tde, sector 75145832
[247387.053233] print_req_error: I/O error, dev tde, sector 75145920
[247387.053282] print_req_error: I/O error, dev tde, sector 75146008
[248010.875068] block tdd: sector-size: 512/512 capacity: 251658240
[248010.913350] block tde: sector-size: 512/512 capacity: 178257920
[249059.521257] print_req_error: 86 callbacks suppressed
[249059.521258] print_req_error: I/O error, dev tde, sector 74633216
[249059.521309] print_req_error: I/O error, dev tde, sector 74633304
[249059.521354] print_req_error: I/O error, dev tde, sector 74633392
[249059.521399] print_req_error: I/O error, dev tde, sector 74633480
[249059.521443] print_req_error: I/O error, dev tde, sector 74633568
[249059.521647] print_req_error: I/O error, dev tde, sector 74633656
[249059.521713] print_req_error: I/O error, dev tde, sector 74633744
[249059.521737] print_req_error: I/O error, dev tde, sector 74633832
[249059.521737] print_req_error: I/O error, dev tde, sector 74633920
[249059.521737] print_req_error: I/O error, dev tde, sector 74634008
[249811.475374] block tdd: sector-size: 512/512 capacity: 251658240
[249811.519873] block tde: sector-size: 512/512 capacity: 178257920
[250899.872276] print_req_error: 86 callbacks suppressed
[250899.872277] print_req_error: I/O error, dev tde, sector 74674176
[250899.872329] print_req_error: I/O error, dev tde, sector 74674264
[250899.872379] print_req_error: I/O error, dev tde, sector 74674352
[250899.872424] print_req_error: I/O error, dev tde, sector 74674440
[250899.872472] print_req_error: I/O error, dev tde, sector 74674528
[250899.872516] print_req_error: I/O error, dev tde, sector 74674616
[250899.872560] print_req_error: I/O error, dev tde, sector 74674704
[250899.872603] print_req_error: I/O error, dev tde, sector 74674792
[250899.872646] print_req_error: I/O error, dev tde, sector 74674880
[250899.872689] print_req_error: I/O error, dev tde, sector 74674968
[251616.965255] block tdd: sector-size: 512/512 capacity: 251658240
[251617.009656] block tde: sector-size: 512/512 capacity: 178257920

Output of xl dmesg on same host:

 \ \/ /___ _ __   | || |  / |___ / | ___|
  \  // _ \ '_ \  | || |_ | | |_ \ |___ \
  /  \  __/ | | | |__   _|| |___) | ___) |
 /_/\_\___|_| |_|    |_|(_)_|____(_)____/

(XEN) [000000299f9da931] Xen version 4.13.5-9.30 (mockbuild@[unknown]) (gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)) debug=n  Tue Mar 21 13:25:35 CET 2023
(XEN) [000000299f9db8ab] Latest ChangeSet: 708e83f0e7d1, pq c81a2963b08d
(XEN) [000000299f9dcc95] build-id: f5d6abf4ba12c46c496f6d910114c239298148bf
(XEN) [000000299f9dd548] Bootloader: GRUB 2.02
(XEN) [000000299f9dde9e] Command line: dom0_mem=1440M,max:1440M watchdog ucode=scan dom0_max_vcpus=1-4 crashkernel=256M,below=4G console=vga vga=mode-0x0311
(XEN) [000000299f9decb1] Xen image load base address: 0xda800000
(XEN) [000000299f9df44e] Video information:
(XEN) [000000299f9dff64]  VGA is graphics mode 640x480, 16 bpp
(XEN) [000000299f9e0a94]  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) [000000299f9e150b] Disc information:
(XEN) [000000299f9e1cc6]  Found 1 MBR signatures
(XEN) [000000299f9e2493]  Found 2 EDD information structures
(XEN) [000000299f9eb3b6] Xen-e820 RAM map:
(XEN) [000000299f9ec0b1]  0000000000000000 - 000000000009dc00 (usable)
(XEN) [000000299f9ecdde]  000000000009dc00 - 00000000000a0000 (reserved)
(XEN) [000000299f9edb15]  00000000000e0000 - 0000000000100000 (reserved)
(XEN) [000000299f9ee817]  0000000000100000 - 00000000d4c60000 (usable)
(XEN) [000000299f9ef4b1]  00000000d4c60000 - 00000000d4c61000 (ACPI NVS)
(XEN) [000000299f9f010e]  00000000d4c61000 - 00000000d4c62000 (reserved)
(XEN) [000000299f9f0c4e]  00000000d4c62000 - 00000000daf48000 (usable)
(XEN) [000000299f9f174e]  00000000daf48000 - 00000000dc1ba000 (reserved)
(XEN) [000000299f9f2236]  00000000dc1ba000 - 00000000dc244000 (ACPI data)
(XEN) [000000299f9f2e85]  00000000dc244000 - 00000000dca1e000 (ACPI NVS)
(XEN) [000000299f9f3981]  00000000dca1e000 - 00000000dcf44000 (reserved)
(XEN) [000000299f9f4439]  00000000dcf44000 - 00000000dd000000 (usable)
(XEN) [000000299f9f4f02]  00000000dd000000 - 00000000e0000000 (reserved)
(XEN) [000000299f9f59e8]  00000000f8000000 - 00000000fc000000 (reserved)
(XEN) [000000299f9f6483]  00000000fe000000 - 00000000fe011000 (reserved)
(XEN) [000000299f9f6fcc]  00000000fec00000 - 00000000fec01000 (reserved)
(XEN) [000000299f9f7b07]  00000000fed00000 - 00000000fed01000 (reserved)
(XEN) [000000299f9f85bc]  00000000fee00000 - 00000000fee01000 (reserved)
(XEN) [000000299f9f9174]  00000000ff000000 - 0000000100000000 (reserved)
(XEN) [000000299f9f9d92]  0000000100000000 - 000000081e000000 (usable)
(XEN) [00000029a1ebb587] Kdump: 256MB (262144kB) at 0xc4c00000
(XEN) [00000029a1f2bb4c] ACPI: RSDP 000F05B0, 0024 (r2 LENOVO)
(XEN) [00000029a1f2d902] ACPI: XSDT DC1D50B8, 00EC (r1 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f2fc14] ACPI: FACP DC203650, 0114 (r6 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f31df4] ACPI: DSDT DC1D5230, 2E41B (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f334e7] ACPI: FACS DC9EDC40, 0040
(XEN) [00000029a1f343bf] ACPI: APIC DC203768, 00BC (r3 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f356df] ACPI: FPDT DC203828, 0044 (r1 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f36965] ACPI: MCFG DC203870, 003C (r1 LENOVO TC-M1A       1300 MSFT       97)
(XEN) [00000029a1f37bbe] ACPI: SSDT DC2038B0, 03BC (r1 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f38fda] ACPI: FIDT DC203C70, 009C (r1 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f3a224] ACPI: SLIC DC203D10, 0176 (r1 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f3b537] ACPI: MSDM DC203E88, 0055 (r3 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f3c74b] ACPI: SSDT DC203EE0, 3159 (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f3db45] ACPI: SSDT DC207040, 26E8 (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f3f1bc] ACPI: HPET DC209728, 0038 (r1 LENOVO TC-M1A       1300 MSFT       5F)
(XEN) [00000029a1f403ab] ACPI: SSDT DC209760, 0C59 (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f417fb] ACPI: UEFI DC20A3C0, 0042 (r1 LENOVO TC-M1A       1300       1000013)
(XEN) [00000029a1f42b40] ACPI: SSDT DC20A408, 0EDE (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f43f15] ACPI: LPIT DC20B2E8, 0094 (r1 LENOVO TC-M1A       1300 MSFT       5F)
(XEN) [00000029a1f45106] ACPI: WSMT DC20B380, 0028 (r1 LENOVO TC-M1A       1300 MSFT       5F)
(XEN) [00000029a1f462cb] ACPI: SSDT DC20B3A8, 029F (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f4762a] ACPI: SSDT DC20B648, 3002 (r2 LENOVO TC-M1A       1300 INTL 20160422)
(XEN) [00000029a1f489ca] ACPI: DBGP DC20E650, 0034 (r1 LENOVO TC-M1A       1300 MSFT       5F)
(XEN) [00000029a1f49b9d] ACPI: DBG2 DC20E688, 0054 (r0 LENOVO TC-M1A       1300 MSFT       5F)
(XEN) [00000029a1f4ae1c] ACPI: DMAR DC20E6E0, 00A8 (r1 LENOVO TC-M1A       1300 INTL        1)
(XEN) [00000029a1f4c086] ACPI: TPM2 DC20E788, 0034 (r3 LENOVO TC-M1A       1300 AMI         0)
(XEN) [00000029a1f4d249] ACPI: LUFT DC20E7C0, 349E2 (r1 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f4e755] ACPI: ASF! DC2431A8, 00A0 (r32 LENOVO TC-M1A       1300 TFSM    F4240)
(XEN) [00000029a1f4fb29] ACPI: BGRT DC243248, 0038 (r1 LENOVO TC-M1A       1300 AMI     10013)
(XEN) [00000029a1f97d56] System RAM: 32655MB (33439356kB)
(XEN) [00000029a3ce9023] No NUMA configuration found
(XEN) [00000029a3cea2c2] Faking a node at 0000000000000000-000000081e000000
(XEN) [0000002a14f9c9cd] Domain heap initialised
(XEN) [0000002a31acb2f6] vesafb: framebuffer at 0x00000000e0000000, mapped to 0xffff82c000201000, using 2048k, total 32704k
(XEN) [0000002a31acc32d] vesafb: mode is 640x480x16, linelength=1280, font 8x8
(XEN) [0000002a31acd3a7] vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
(XEN) [0000002a31ae02b7] CPU Vendor: Intel, Family 6 (0x6), Model 94 (0x5e), Stepping 3 (raw 000506e3)
(XEN) [0000002a33af5975] found SMP MP-table at 000fccc0
(XEN) [0000002a33c42e12] SMBIOS 3.0 present.
(XEN) [0000002a33d604d3] Using APIC driver default
(XEN) [0000002a33e92765] XSM Framework v1.0.1 initialized
(XEN) [0000002a33fe47d9] Initialising XSM SILO mode
(XEN) [0000002a3412e2d5] ACPI: PM-Timer IO Port: 0x1808 (24 bits)
(XEN) [0000002a342a651f] ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) [0000002a3443b343] ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0]
(XEN) [0000002a34556c75] ACPI: 32/64X FACS address mismatch in FADT - dc9edc40/0000000000000000, using 32
(XEN) [0000002a346cf1b8] ACPI:             wakeup_vec[dc9edc4c], vec_size[20]
(XEN) [0000002a3489d86a] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) [0000002a34a43d2c] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) [0000002a34bea309] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
(XEN) [0000002a34d9044d] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
(XEN) [0000002a34f36537] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
(XEN) [0000002a35154603] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
(XEN) [0000002a352fabd5] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
(XEN) [0000002a354a1211] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
(XEN) [0000002a3565df9a] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) [0000002a35808be6] ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
(XEN) [0000002a359b39cb] ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
(XEN) [0000002a35b5e1e2] ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
(XEN) [0000002a35d08c67] ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
(XEN) [0000002a35eb33c2] ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
(XEN) [0000002a3605d964] ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
(XEN) [0000002a3620849c] ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
(XEN) [0000002a363cb3e8] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [0000002a36595b80] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
(XEN) [0000002a366cb8b1] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [0000002a367d4db3] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [0000002a368f8431] Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) [0000002a36a8ada3] ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [0000002a36c1dc02] PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) [0000002a36d59445] PCI: MCFG area at f8000000 reserved in E820
(XEN) [0000002a36edfe08] PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) [0000002a37096b04] ACPI: BGRT: invalidating v1 image at 0xd7746018
(XEN) [0000002a3722e1f6] Using ACPI (MADT) for SMP configuration information
(XEN) [0000002a373d810e] SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) [0000002a375583e1] IRQ limits: 120 GSI, 1432 MSI/MSI-X
(XEN) [0000002a386c7828] Switched to APIC driver x2apic_phys
(XEN) [0000002a39621600] microcode: CPU0 updated from revision 0xc2 to 0xf0, date = 2021-11-12
(XEN) [0000002a3976c4a9] xstate: size: 0x440 and states: 0x1f
(XEN) [0000002a398e1990] CPU0: Intel machine check reporting enabled
(XEN) [0000002a39a6c574] Speculative mitigation facilities:
(XEN) [0000002a39bc5c79]   Hardware hints: RSBA
(XEN) [0000002a39ce7e07]   Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH MD_CLEAR SRBDS_CTRL
(XEN) [0000002a39e33c78]   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) [0000002a39fdb7c2]   Xen settings: BTI-Thunk JMP, SPEC_CTRL: IBRS+ STIBP+ SSBD- PSFD-, Other: SRB_LOCK+ IBPB-ctxt L1D_FLUSH VERW BRANCH_HARDEN
(XEN) [0000002a3a216953]   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 39, Safe address 8000000000
(XEN) [0000002a3ea3b3f4]   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR
(XEN) [0000002a432636ba]   Support for PV VMs: MSR_SPEC_CTRL EAGER_FPU MD_CLEAR
(XEN) [0000002a4576a629]   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (with PCID)
(XEN) [0000002a4a097425]   PV L1TF shadowing: Dom0 disabled, DomU enabled
(XEN) [0000002a4c610011] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [0000002a57199776] Platform timer is 23.999MHz HPET
(XEN) [    7.506401] Detected 2808.027 MHz processor.
(XEN) [    7.517967] alt table ffff82d080456ed0 -> ffff82d080465d40
(XEN) [    7.558340] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    7.566526] Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    7.574524] Intel VT-d Snoop Control not enabled.
(XEN) [    7.582603] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    7.590514] Intel VT-d Queued Invalidation enabled.
(XEN) [    7.598541] Intel VT-d Interrupt Remapping enabled.
(XEN) [    7.606462] Intel VT-d Posted Interrupt not enabled.
(XEN) [    7.614484] Intel VT-d Shared EPT tables enabled.
(XEN) [    7.622943] I/O virtualisation enabled
(XEN) [    7.630834]  - Dom0 mode: Relaxed
(XEN) [    7.638361] Interrupt remapping enabled
(XEN) [    7.646072] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    7.664154] ENABLING IO-APIC IRQs
(XEN) [    7.671572]  -> Using old ACK method
(XEN) [    7.679731] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    8.731232] Allocated console ring of 64 KiB.
(XEN) [    8.738655] VMX: Supported advanced features:
(XEN) [    8.746157]  - APIC MMIO access virtualisation
(XEN) [    8.753474]  - APIC TPR shadow
(XEN) [    8.760670]  - Extended Page Tables (EPT)
(XEN) [    8.767984]  - Virtual-Processor Identifiers (VPID)
(XEN) [    8.775270]  - Virtual NMI
(XEN) [    8.782564]  - MSR direct-access bitmap
(XEN) [    8.789819]  - Unrestricted Guest
(XEN) [    8.797141]  - VMCS shadowing
(XEN) [    8.804377]  - VM Functions
(XEN) [    8.811691]  - Virtualisation Exceptions
(XEN) [    8.818731]  - Page Modification Logging
(XEN) [    8.825709] HVM: ASIDs enabled.
(XEN) [    8.832982] HVM: VMX enabled
(XEN) [    8.840043] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    8.847293] HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) [    8.854603] alt table ffff82d080456ed0 -> ffff82d080465d40
(XEN) [0000002b3bc37dd3] microcode: CPU2 updated from revision 0xc2 to 0xf0, date = 2021-11-12
(XEN) [0000002b3e73fd42] microcode: CPU4 updated from revision 0xc2 to 0xf0, date = 2021-11-12
(XEN) [0000002b411f0901] microcode: CPU6 updated from revision 0xc2 to 0xf0, date = 2021-11-12
(XEN) [    8.910909] Brought up 8 CPUs
(XEN) [    8.920420] Testing NMI watchdog on all CPUs: ok
(XEN) [    8.960783] Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) [    8.990587] mcheck_poll: Machine check polling timer started.
(XEN) [    9.000212] Dom0 has maximum 888 PIRQs
(XEN) [    9.009747] csched_alloc_domdata: setting dom 0 as the privileged domain
(XEN) [    9.028512] NX (Execute Disable) protection active
(XEN) [    9.037803] *** Building a PV Dom0 ***
(XEN) [    9.360826]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    9.370255]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x302c000
(XEN) [    9.389229] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    9.398543]  Dom0 alloc.:   0000000800000000->0000000804000000 (347002 pages to be allocated)
(XEN) [    9.417022]  Init. ramdisk: 000000081cb7a000->000000081dfff214
(XEN) [    9.426481] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    9.435756]  Loaded kernel: ffffffff81000000->ffffffff8302c000
(XEN) [    9.445126]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    9.454402]  Phys-Mach map: 0000008000000000->00000080002d0000
(XEN) [    9.463851]  Start info:    ffffffff8302c000->ffffffff8302c4b8
(XEN) [    9.473127]  Xenstore ring: 0000000000000000->0000000000000000
(XEN) [    9.482765]  Console ring:  0000000000000000->0000000000000000
(XEN) [    9.492060]  Page tables:   ffffffff8302d000->ffffffff8304a000
(XEN) [    9.501535]  Boot stack:    ffffffff8304a000->ffffffff8304b000
(XEN) [    9.510888]  TOTAL:         ffffffff80000000->ffffffff83400000
(XEN) [    9.520426]  ENTRY ADDRESS: ffffffff8242b180
(XEN) [    9.531941] Dom0 has maximum 4 VCPUs
(XEN) [    9.563237] Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [   11.555503] Initial low memory virq threshold set at 0x4000 pages.
(XEN) [   11.562838] Scrubbing Free RAM in background
(XEN) [   11.569731] Std. Loglevel: Errors, warnings and info
(XEN) [   11.576733] Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) [   11.584183] ***************************************************
(XEN) [   11.591265] Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
(XEN) [   11.598532] enabled.  Please assess your configuration and choose an
(XEN) [   11.605681] explicit 'smt=<bool>' setting.  See XSA-273.
(XEN) [   11.612972] ***************************************************
(XEN) [   11.620097] Booted on MLPDS/MFBDS-vulnerable hardware with SMT/Hyperthreading
(XEN) [   11.634350] enabled.  Mitigations will not be fully effective.  Please
(XEN) [   11.641605] choose an explicit smt=<bool> setting.  See XSA-297.
(XEN) [   11.649048] ***************************************************
(XEN) [   11.656345] 3... 2... 1...
(XEN) [   14.665205] Xen is relinquishing VGA console.
(XEN) [   14.674950] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) [   14.675033] Freed 608kB init memory
(XEN) [   16.595699] Bogus DMIBAR 0xfed18001 on 0000:00:00.0

May be the culprit?

[250899.872516] print_req_error: I/O error, dev tde, sector 74674616

Thanks