XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    VM UUID via dmidecode does not match VM ID in xen-orchestra

    Scheduled Pinned Locked Moved Infrastructure as Code
    23 Posts 5 Posters 445 Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • CyrilleC Offline
      Cyrille Vates 🪐 DevOps Team
      last edited by

      Hi,
      Do you know what causes the system UUID to change and not match the VM UUID?

      In the tests I have run (with Debian and Microk8s), it never changed.

      D 1 Reply Last reply Reply Quote 0
      • D Offline
        deefdragon @Cyrille
        last edited by deefdragon

        @Cyrille Genuinly no idea unfortunately. The only thing I can think that I've done is that the VMs are being loaded in from a template.

        I had to deploy new VMs through my terraform recently, and I AM still seeing an incorrect system UUID on those, so I can only guess its somehow the template?

        I am also seeing the serial number from DMIDECODE match the VM ID in xen orchestra as well still.

        CyrilleC 1 Reply Last reply Reply Quote 0
        • CyrilleC Offline
          Cyrille Vates 🪐 DevOps Team @deefdragon
          last edited by

          @deefdragon can you check if /sys/hypervisor/uuid matches your VMs UUID?

          D 1 Reply Last reply Reply Quote 0
          • D Offline
            deefdragon @Cyrille
            last edited by

            @Cyrille I am seeing /sys/hypervisor/uuid matching dmidecode's Serial Number matching VM ID in xen orchestra.

            dmidecode's UUID however is different from those three.

            D 1 Reply Last reply Reply Quote 1
            • D Offline
              dinhngtu Vates 🪐 XCP-ng Team @deefdragon
              last edited by dinhngtu

              @deefdragon All three values match for me with XCP-ng 8.3, Ubuntu 24.04 guests and kernel 6.8:

              $ sudo cat /sys/devices/virtual/dmi/id/product_serial
adc9b6ba-d187-0844-13a6-5f1dc155bf6e
$ sudo cat /sys/devices/virtual/dmi/id/product_uuid
adc9b6ba-d187-0844-13a6-5f1dc155bf6e
$ sudo cat /sys/hypervisor/uuid
adc9b6ba-d187-0844-13a6-5f1dc155bf6e

              From your Github issue, it looks like a confusion in byte order between the System UUID (6a87cb0f-ca4c-ffa5-3ca2-fc398fb25eac) and the XCP-ng VM UUID (0fcb876a-4cca-a5ff-3ca2-fc398fb25eac).

              I'd check the guest kernel version first.

              D 1 Reply Last reply Reply Quote 0
              • D Offline
                deefdragon @dinhngtu
                last edited by deefdragon

                @dinhngtu

                Ah, I see the pattern now. That is bizarre.

                deef@k31-w-3bfbbe:~$ sudo cat /sys/devices/virtual/dmi/id/product_serial
0b08f477-491a-a982-23c4-d224723624ea
deef@k31-w-3bfbbe:~$ sudo cat /sys/devices/virtual/dmi/id/product_uuid
77f4080b-1a49-82a9-23c4-d224723624ea
deef@k31-w-3bfbbe:~$ sudo cat /sys/hypervisor/uuid
0b08f477-491a-a982-23c4-d224723624ea

                Heres the output from a series of commands I ran to get the info of the worker. Seeing 6.8 and 24.04 myself.

                deef@k31-w-3bfbbe:~$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

deef@k31-w-3bfbbe:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.2 LTS
Release:	24.04
Codename:	noble

deef@k31-w-3bfbbe:~$ hostnamectl
 Static hostname: k31-w-3bfbbe
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: 9bae21fe1e10d9da92da72599fd7b4a3
         Boot ID: 959ebf2dc51e403eaf00103d51e8397e
  Virtualization: xen
Operating System: Ubuntu 24.04.2 LTS              
          Kernel: Linux 6.8.0-64-generic
    Architecture: x86-64
 Hardware Vendor: Xen
  Hardware Model: HVM domU
Firmware Version: 4.17
   Firmware Date: Tue 2025-05-13
    Firmware Age: 2month 2w 3d

deef@k31-w-3bfbbe:~$ uname -r
6.8.0-64-generic

                LMK if there is anything else you want me to throw at it, or any other debug info you want me to get. I'm out of my depth when it comes to debugging this kind of thing.

                D 1 Reply Last reply Reply Quote 0
                • D Offline
                  dinhngtu Vates 🪐 XCP-ng Team @deefdragon
                  last edited by

                  @deefdragon How about host Xen and XAPI versions? I'm on the bleeding edge, so this may have been fixed somewhere already.

                  # unbuffer yum list installed | grep xen\\\|xapi
mellanox-mlnxen.x86_64            5.9_0.5.5.0-2.1.xcpng8.3    @xcp:main/$releasever
python2-xapi-storage.x86_64       24.19.2-1.10.xcpng8.3       @xcp-ng-testing
xapi-core.x86_64                  25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xapi-nbd.x86_64                   25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xapi-rrd2csv.x86_64               25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xapi-storage-script.x86_64        25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xapi-tests.x86_64                 25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xapi-xe.x86_64                    25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xcp-ng-xapi-plugins.noarch        1.12.0-1.xcpng8.3           @xcp-ng-testing
xen-crashdump-analyser.x86_64     2.6.1-1.xcpng8.3            @xcp:main/$releasever
xen-dom0-libs.x86_64              4.17.5-15.2.xcpng8.3        @xcp-ng-incoming
xen-dom0-tools.x86_64             4.17.5-15.2.xcpng8.3        @xcp-ng-incoming
xen-hypervisor.x86_64             4.17.5-15.2.xcpng8.3        @xcp-ng-incoming
xen-libs.x86_64                   4.17.5-15.2.xcpng8.3        @xcp-ng-incoming
xen-tools.x86_64                  4.17.5-15.2.xcpng8.3        @xcp-ng-incoming
xengt-userspace.noarch            4.0.0-1.xcpng8.3            @xcp:main/$releasever
xenopsd.x86_64                    25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xenopsd-cli.x86_64                25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xenopsd-xc.x86_64                 25.6.0-1.10.xcpng8.3        @xcp-ng-incoming
xenserver-dracut.noarch           10-2.xcpng8.3               @xcp:main/$releasever
xenserver-hwdata.noarch           20240411-1.xcpng8.3         @xcp:main/$releasever
xenserver-status-report.noarch    2.0.11-1.xcpng8.3           @xcp-ng-ci
                  D 1 Reply Last reply Reply Quote 0
                  • D Offline
                    deefdragon @dinhngtu
                    last edited by

                    @dinhngtu

                    Only difference looks to be xen-dom0 etc. on 4.17.5-13.1.xcpng8.3 vs your 4.17.5-15.2.xcpng8.3

                    # unbuffer yum list installed | grep xen\\\|xapi
mellanox-mlnxen.x86_64                   5.9_0.5.5.0-2.1.xcpng8.3         @xcp:main/$releasever
python2-xapi-storage.x86_64              24.19.2-1.10.xcpng8.3            @xcp-ng-updates      
xapi-core.x86_64                         25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xapi-nbd.x86_64                          25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xapi-rrd2csv.x86_64                      25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xapi-storage-script.x86_64               25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xapi-tests.x86_64                        25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xapi-xe.x86_64                           25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xcp-ng-xapi-plugins.noarch               1.12.0-1.xcpng8.3                @xcp-ng-updates      
xen-crashdump-analyser.x86_64            2.6.1-1.xcpng8.3                 @xcp:main/$releasever
xen-dom0-libs.x86_64                     4.17.5-13.1.xcpng8.3             @xcp-ng-updates      
xen-dom0-tools.x86_64                    4.17.5-13.1.xcpng8.3             @xcp-ng-updates      
xen-hypervisor.x86_64                    4.17.5-13.1.xcpng8.3             @xcp-ng-updates      
xen-libs.x86_64                          4.17.5-13.1.xcpng8.3             @xcp-ng-updates      
xen-tools.x86_64                         4.17.5-13.1.xcpng8.3             @xcp-ng-updates      
xengt-userspace.noarch                   4.0.0-1.xcpng8.3                 @xcp:main/$releasever
xenopsd.x86_64                           25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xenopsd-cli.x86_64                       25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xenopsd-xc.x86_64                        25.6.0-1.5.xcpng8.3              @xcp-ng-updates      
xenserver-dracut.noarch                  10-2.xcpng8.3                    @xcp:main/$releasever
xenserver-hwdata.noarch                  20240411-1.xcpng8.3              @xcp:main/$releasever
xenserver-status-report.noarch           2.0.11-1.xcpng8.3                @xcp-ng-updates
                    D 1 Reply Last reply Reply Quote 0
                    • D Offline
                      dinhngtu Vates 🪐 XCP-ng Team @deefdragon
                      last edited by

                      @deefdragon Your XAPI is also lagging behind (25.6.0-1.5 vs latest 25.6.0-1.9). Could you update your pool?

                      D 1 Reply Last reply Reply Quote 0
                      • TeddyAstieT Offline
                        TeddyAstie Vates 🪐 XCP-ng Team Xen Guru
                        last edited by 

                        deef@k31-w-3bfbbe:~$ sudo cat /sys/devices/virtual/dmi/id/product_serial
0b08f477-491a-a982-23c4-d224723624ea
deef@k31-w-3bfbbe:~$ sudo cat /sys/devices/virtual/dmi/id/product_uuid
77f4080b-1a49-82a9-23c4-d224723624ea
deef@k31-w-3bfbbe:~$ sudo cat /sys/hypervisor/uuid
0b08f477-491a-a982-23c4-d224723624ea

                        It looks like a endianness issue (0b08f477 vs 77f4080b).

                        1 Reply Last reply Reply Quote 0
                        • TeddyAstieT Offline
                          TeddyAstie Vates 🪐 XCP-ng Team Xen Guru
                          last edited by

                          @deefdragon can you provide us the output of dmidecode in the guest ?

                          1 Reply Last reply Reply Quote 0
                          • D Offline
                            deefdragon @dinhngtu
                            last edited by

                            @dinhngtu Updated through xo. These are the current values.

                            mellanox-mlnxen.x86_64               5.9_0.5.5.0-2.1.xcpng8.3       @xcp:main/$releasever
python2-xapi-storage.x86_64          24.19.2-1.10.xcpng8.3          @xcp-ng-updates      
xapi-core.x86_64                     25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xapi-nbd.x86_64                      25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xapi-rrd2csv.x86_64                  25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xapi-storage-script.x86_64           25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xapi-tests.x86_64                    25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xapi-xe.x86_64                       25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xcp-ng-xapi-plugins.noarch           1.12.0-1.xcpng8.3              @xcp-ng-updates      
xen-crashdump-analyser.x86_64        2.6.1-1.xcpng8.3               @xcp:main/$releasever
xen-dom0-libs.x86_64                 4.17.5-15.1.xcpng8.3           @xcp-ng-updates      
xen-dom0-tools.x86_64                4.17.5-15.1.xcpng8.3           @xcp-ng-updates      
xen-hypervisor.x86_64                4.17.5-15.1.xcpng8.3           @xcp-ng-updates      
xen-libs.x86_64                      4.17.5-15.1.xcpng8.3           @xcp-ng-updates      
xen-tools.x86_64                     4.17.5-15.1.xcpng8.3           @xcp-ng-updates      
xengt-userspace.noarch               4.0.0-1.xcpng8.3               @xcp:main/$releasever
xenopsd.x86_64                       25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xenopsd-cli.x86_64                   25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xenopsd-xc.x86_64                    25.6.0-1.9.xcpng8.3            @xcp-ng-updates      
xenserver-dracut.noarch              10-2.xcpng8.3                  @xcp:main/$releasever
xenserver-hwdata.noarch              20240411-1.xcpng8.3            @xcp:main/$releasever
xenserver-status-report.noarch       2.0.11-1.xcpng8.3              @xcp-ng-updates

                            Gave one of the VMs a reboot and got the same result as previous.

                            Full DMI Decode as requested @TeddyAstie

                            # dmidecode
# dmidecode 3.5
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
19 structures occupying 691 bytes.
Table at 0xEE6FC000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: Xen
	Version: 4.17
	Release Date: 07/10/2025
	Address: 0xE8000
	Runtime Size: 96 kB
	ROM Size: 64 kB
	Characteristics:
		PCI is supported
		EDD is supported
		Targeted content distribution is supported
	BIOS Revision: 4.17

Handle 0x0100, DMI type 1, 27 bytes
System Information
	Manufacturer: Xen
	Product Name: HVM domU
	Version: 4.17
	Serial Number: 0b08f477-491a-a982-23c4-d224723624ea
	UUID: 77f4080b-1a49-82a9-23c4-d224723624ea
	Wake-up Type: Power Switch
	SKU Number: Not Specified
	Family: Not Specified

Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
	Manufacturer: Xen
	Type: Other
	Lock: Not Present
	Version: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Boot-up State: Safe
	Power Supply State: Safe
	Thermal State: Safe
	Security Status: Unknown
	OEM Information: 0x00000000
	Height: Unspecified
	Number Of Power Cords: Unspecified
	Contained Elements: 0

Handle 0x0401, DMI type 4, 35 bytes
Processor Information
	Socket Designation: CPU 1
	Type: Central Processor
	Family: Other
	Manufacturer: Intel
	ID: F1 06 04 00 FF FB CB 0F
	Version: Not Specified
	Voltage: Unknown
	External Clock: Unknown
	Max Speed: 2100 MHz
	Current Speed: 2100 MHz
	Status: Populated, Enabled
	Upgrade: Other
	L1 Cache Handle: Not Provided
	L2 Cache Handle: Not Provided
	L3 Cache Handle: Not Provided
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x0402, DMI type 4, 35 bytes
Processor Information
	Socket Designation: CPU 2
	Type: Central Processor
	Family: Other
	Manufacturer: Intel
	ID: F1 06 04 00 FF FB CB 0F
	Version: Not Specified
	Voltage: Unknown
	External Clock: Unknown
	Max Speed: 2100 MHz
	Current Speed: 2100 MHz
	Status: Populated, Enabled
	Upgrade: Other
	L1 Cache Handle: Not Provided
	L2 Cache Handle: Not Provided
	L3 Cache Handle: Not Provided
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x0403, DMI type 4, 35 bytes
Processor Information
	Socket Designation: CPU 3
	Type: Central Processor
	Family: Other
	Manufacturer: Intel
	ID: F1 06 04 00 FF FB CB 0F
	Version: Not Specified
	Voltage: Unknown
	External Clock: Unknown
	Max Speed: 2100 MHz
	Current Speed: 2100 MHz
	Status: Populated, Enabled
	Upgrade: Other
	L1 Cache Handle: Not Provided
	L2 Cache Handle: Not Provided
	L3 Cache Handle: Not Provided
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x0404, DMI type 4, 35 bytes
Processor Information
	Socket Designation: CPU 4
	Type: Central Processor
	Family: Other
	Manufacturer: Intel
	ID: F1 06 04 00 FF FB CB 0F
	Version: Not Specified
	Voltage: Unknown
	External Clock: Unknown
	Max Speed: 2100 MHz
	Current Speed: 2100 MHz
	Status: Populated, Enabled
	Upgrade: Other
	L1 Cache Handle: Not Provided
	L2 Cache Handle: Not Provided
	L3 Cache Handle: Not Provided
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x0405, DMI type 4, 35 bytes
Processor Information
	Socket Designation: CPU 5
	Type: Central Processor
	Family: Other
	Manufacturer: Intel
	ID: F1 06 04 00 FF FB CB 0F
	Version: Not Specified
	Voltage: Unknown
	External Clock: Unknown
	Max Speed: 2100 MHz
	Current Speed: 2100 MHz
	Status: Populated, Enabled
	Upgrade: Other
	L1 Cache Handle: Not Provided
	L2 Cache Handle: Not Provided
	L3 Cache Handle: Not Provided
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x0406, DMI type 4, 35 bytes
Processor Information
	Socket Designation: CPU 6
	Type: Central Processor
	Family: Other
	Manufacturer: Intel
	ID: F1 06 04 00 FF FB CB 0F
	Version: Not Specified
	Voltage: Unknown
	External Clock: Unknown
	Max Speed: 2100 MHz
	Current Speed: 2100 MHz
	Status: Populated, Enabled
	Upgrade: Other
	L1 Cache Handle: Not Provided
	L2 Cache Handle: Not Provided
	L3 Cache Handle: Not Provided
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x0B00, DMI type 11, 5 bytes
OEM Strings
	String 1: Xen
	String 2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d

Handle 0x1000, DMI type 16, 15 bytes
Physical Memory Array
	Location: Other
	Use: System Memory
	Error Correction Type: Multi-bit ECC
	Maximum Capacity: 24568 MB
	Error Information Handle: Not Provided
	Number Of Devices: 2

Handle 0x1100, DMI type 17, 27 bytes
Memory Device
	Array Handle: 0x1000
	Error Information Handle: 0x0000
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 16 GB
	Form Factor: DIMM
	Set: None
	Locator: DIMM 0
	Bank Locator: Not Specified
	Type: RAM
	Type Detail: None
	Speed: Unknown
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x1300, DMI type 19, 15 bytes
Memory Array Mapped Address
	Starting Address: 0x00000000000
	Ending Address: 0x003FFFFFFFF
	Range Size: 16 GB
	Physical Array Handle: 0x1000
	Partition Width: 1

Handle 0x1400, DMI type 20, 19 bytes
Memory Device Mapped Address
	Starting Address: 0x00000000000
	Ending Address: 0x003FFFFFFFF
	Range Size: 16 GB
	Physical Device Handle: 0x1100
	Memory Array Mapped Address Handle: 0x1300
	Partition Row Position: 1

Handle 0x1101, DMI type 17, 27 bytes
Memory Device
	Array Handle: 0x1000
	Error Information Handle: 0x0000
	Total Width: 64 bits
	Data Width: 64 bits
	Size: 8184 MB
	Form Factor: DIMM
	Set: None
	Locator: DIMM 1
	Bank Locator: Not Specified
	Type: RAM
	Type Detail: None
	Speed: Unknown
	Manufacturer: Not Specified
	Serial Number: Not Specified
	Asset Tag: Not Specified
	Part Number: Not Specified

Handle 0x1301, DMI type 19, 15 bytes
Memory Array Mapped Address
	Starting Address: 0x00400000000
	Ending Address: 0x005FF7FFFFF
	Range Size: 8184 MB
	Physical Array Handle: 0x1000
	Partition Width: 1

Handle 0x1401, DMI type 20, 19 bytes
Memory Device Mapped Address
	Starting Address: 0x00400000000
	Ending Address: 0x005FF7FFFFF
	Range Size: 8184 MB
	Physical Device Handle: 0x1101
	Memory Array Mapped Address Handle: 0x1301
	Partition Row Position: 1

Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
	Status: No errors detected

Handle 0xFEFF, DMI type 127, 4 bytes
End Of Table
                            1 Reply Last reply Reply Quote 0
                            • D Offline
                              deefdragon
                              last edited by

                              Out of curiosity, I dumped the DMI into a bin and opened it up in a hex editor.

                              I am seeing ASCII of the ID, but also a variant encoded in binary. In both cases, its formatted as 0b08f477-491a-a982-23c4-d224723624ea.

                              I believe the ASCII version is the one that gets populated into the serial number as it comes after ASCII encoded versions of the 3 lines above it in the decode.

                              TeddyAstieT 1 Reply Last reply Reply Quote 0
                              • TeddyAstieT Offline
                                TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @deefdragon
                                last edited by TeddyAstie

                                @deefdragon said in VM UUID via dmidecode does not match VM ID in xen-orchestra:

                                Out of curiosity, I dumped the DMI into a bin and opened it up in a hex editor.

                                I am seeing ASCII of the ID, but also a variant encoded in binary. In both cases, its formatted as 0b08f477-491a-a982-23c4-d224723624ea.

                                I believe the ASCII version is the one that gets populated into the serial number as it comes after ASCII encoded versions of the 3 lines above it in the decode.

                                In SMBIOS 2.8, the UUID is supposed to be encoded in little endian (i.e Microsoft GUID). Yet it is put as big endian instead. So when Linux generates the UUID string from the SMBIOS table, it is considered as little endian which causes this mismatch.

                                SMBIOS 2.4 is supposed to be used (which appears to be using big endian UUIDs), but for some reason, something in XCP-ng UEFI supports forces it to be SMBIOS 2.8.

                                So the binary UUID is the same, just that it is interpreted with a different endianness due to accidental format change.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post