Slow Random R/W From VMWare Import
-
Hello All!
Closing in on our move over to XCP from ESXi but we are having (hopefully) one last stumble. Our software vendor gave us their golden image of a 2022 Windows Server but when we do an import of that working ESXi image over to XCP-NG, the random r/w absolutely tank while sequential is fine. A fresh copy of W22 installed on the same iSCSI SR (with the same CPU/RAM configuration) performs as expected. I uninstalled the VM tools once it got over to XCP, rebooted, installed Xen Tools and rebooted once more before running the test. Any ideas what could make this drop so much?
-
Hi!
So on the same configuration/hardware, 2x VMs with the same OS are performing very differently, that's right?
-
@olivierlambert Yessir! I made sure to move them on the same host to make sure nothing funny could happen there. Only difference, at least on the surface, is one was imported and one was installed fresh from ISO.
Additional info from continuing to poke at it:
From watching task manager while CrystalDiskMark is running, both are similar. Sequential XCP installed one is about 40% on a single core while the VMW is 50%ish. Both are pegged on that single core with the random r/w but it seems the ceiling for that VMW one is much lower. -
@meteora An interesting thing to try would be switching the disks attached to the two VMs and then retest. This might help identify an underlying issue tied to the template being used with each VM.
-
@Danp Here are the results of that below. It definitely picks up with the XCP VM having the VMW import VM disk (but not the other way). This gets me close but still a bit off in terms of the raw IOPS. Assuming whatever the templates do are helping in this situation?
One odd thing I noticed when i swapped the disk to on VM from the other, had to reinstall the XenServer 9.3.2 tools to get them function again. While they were down i was getting 2.1-2.2GB/s up from the 1.6-1.8GB/s sequential on both read and writes, which is by far the most I’ve been able to squeeze out of XCP-ng. Random r/w dropped down to 50MB/s which is even worse than the VM import. Not sure that has anything to do with this but a weird thing I wanted to note too.
-
Let's check the result of
xe vm-param-list uuid=<VM UUID>for both VMs so we could compare. -
@olivierlambert
VM Import:uuid ( RO) : 10edb451-487d-3ed5-2b99-59bc6bd6c42a name-label ( RW): VM-IMPORT name-description ( RW): user-version ( RW): 1 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): running memory-actual ( RO): 17181958144 memory-target ( RO): 0 memory-overhead ( RO): 140509184 memory-static-max ( RW): 17179869184 memory-dynamic-max ( RW): 17179869184 memory-dynamic-min ( RW): 17179869184 memory-static-min ( RW): 4294967296 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): 34c0b27e-9ef6-fa0a-a4b3-dbcf480ab5bb hvm ( RO): true platform (MRW): timeoffset: -18000; cores-per-socket: 2; device-model: qemu-upstream-uefi; nx: true; acpi: 1; apic: true; pae: true; hpet: true; viridian: true allowed-operations (SRO): changing_dynamic_range; migrate_send; pool_migrate; changing_VCPUs_live; suspend; hard_reboot; hard_shutdown; clean_reboot; clean_shutdown; pause; checkpoint; snapshot current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): acd5cb1f-c846-4b60-b8da-43e37a2688d5; eb38d624-e57e-4887-8cb1-c1477356f4bf domain-type ( RW): hvm current-domain-type ( RO): hvm HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): firmware: uefi; order: cdn HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-f7fa3223-2c100800-00000121-0000000f-219c0fab-0040070c-00000000-00001000-bc000400-00000000-00000000-00000000-00000001-00000000-00000000-0c0aa163-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): acd5cb1f-c846-4b60-b8da-43e37a2688d5 affinity ( RW): <not in database> other-config (MRW): xo:10edb451: {"creation":{"user":"688d13d3-4908-48e9-ac8a-68e5bf87be07"}}; mac_seed: e7104501-29b2-2ea9-5996-074d16ed87d3; vgpu_pci: ; base_template_name: Other install media; install-methods: cdrom dom-id ( RO): 32 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472" /><restriction field="vcpus-max" max="32" /><restriction property="number-of-vbds" max="255" /><restriction property="number-of-vifs" max="7" /><restriction field="has-vendor-device" value="false" /></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 20240404T02:58:19Z install-time ( RO): 19700101T00:00:00Z VCPUs-number ( RO): 4 VCPUs-utilisation (MRO): 0: 0.079; 1: 0.022; 2: 0.020; 3: 0.020 os-version (MRO): name: Microsoft Windows Server 2022 Standard|C:\Windows|\Device\Harddisk0\Partition4; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0 PV-drivers-version (MRO): major: 9; minor: 3; micro: 2; build: 110 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): b97eebc0-8ace-e023-7542-5bc38a803190 networks (MRO): 0/ip: 10.2.18.8; 0/ipv4/0: 10.2.18.8; 0/ipv6/0: fe80:0000:0000:0000:9a33:e175:9262:cf6f PV-drivers-detected ( RO): true other (MRO): platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-xs-batcmd: 1; feature-wuRebootRequired: 1; feature-updatetime: 1; feature-ts2: 1; feature-ts: 1; feature-suspend: 1; feature-static-ip-setting: 1; feature-shutdown: 1; feature-setcomputername: 1; feature-s4: 1; feature-s3: 1; feature-reboot: 1; feature-poweroff: 1; feature-laptop-slate-mode: 1; feature-balloon: 1; data-ts: 1 live ( RO): true guest-metrics-last-updated ( RO): 20240404T02:58:31Z can-use-hotplug-vbd ( RO): true can-use-hotplug-vif ( RO): true cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 1346807414286446182:79618871952688207 hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3dXCP ISO Install
uuid ( RO) : 07544174-d164-f11a-48b4-55f27d6c99ad name-label ( RW): XCP-W22 name-description ( RW): user-version ( RW): 1 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): running memory-actual ( RO): 17181958144 memory-target ( RO): 0 memory-overhead ( RO): 140509184 memory-static-max ( RW): 17179869184 memory-dynamic-max ( RW): 17179869184 memory-dynamic-min ( RW): 17179869184 memory-static-min ( RW): 1073741824 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): bf0a5af2-ca90-739a-76b0-3c15edf87c9a hvm ( RO): true platform (MRW): timeoffset: -25200; secureboot: false; device-model: qemu-upstream-uefi; videoram: 8; hpet: true; viridian_apic_assist: true; apic: true; device_id: 0002; cores-per-socket: 2; viridian_crash_ctl: true; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian_stimer: true; viridian: true; acpi: 1; viridian_reference_tsc: true allowed-operations (SRO): changing_dynamic_range; migrate_send; pool_migrate; changing_VCPUs_live; suspend; hard_reboot; hard_shutdown; clean_reboot; clean_shutdown; pause; checkpoint; snapshot current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): acd5cb1f-c846-4b60-b8da-43e37a2688d5; eb38d624-e57e-4887-8cb1-c1477356f4bf domain-type ( RW): hvm current-domain-type ( RO): hvm HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): firmware: uefi; order: cd HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-f7fa3223-2c100800-00000121-0000000f-219c0fab-0040070c-00000000-00001000-bc000400-00000000-00000000-00000000-00000001-00000000-00000000-0c0aa163-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): acd5cb1f-c846-4b60-b8da-43e37a2688d5 affinity ( RW): <not in database> other-config (MRW): xo:07544174: {"creation":{"date":"2024-03-27T14:40:02.387Z","template":"1c33af1c-e919-418c-ad45-85d7d6fb604a","user":"688d13d3-4908-48e9-ac8a-68e5bf87be07"}}; base_template_name: Windows Server 2022 (64-bit); import_task: OpaqueRef:bd59858f-d6a4-4f96-bd7b-0c63e84d5bec; mac_seed: c7490bbf-71a9-a855-0d07-0db105c54fc1; install-methods: cdrom dom-id ( RO): 23 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-min" min="2"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction field="supports-bios" value="yes"/><restriction field="supports-uefi" value="yes"/><restriction field="supports-secure-boot" value="yes"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 20240403T22:07:03Z install-time ( RO): 20240327T14:40:02Z VCPUs-number ( RO): 4 VCPUs-utilisation (MRO): 0: 0.025; 1: 0.009; 2: 0.016; 3: 0.013 os-version (MRO): name: Microsoft Windows Server 2022 Standard|C:\Windows|\Device\Harddisk0\Partition3; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0 PV-drivers-version (MRO): major: 9; minor: 3; micro: 2; build: 110 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): ba544fdd-d55b-cc2e-7dea-a9331738154e; 75776d87-041e-1b1e-5fb1-559d6cc3ee8b networks (MRO): 0/ip: 10.2.18.30; 0/ipv4/0: 10.2.18.30; 0/ipv6/0: fe80:0000:0000:0000:dc68:82d6:b4c3:5878 PV-drivers-detected ( RO): true other (MRO): platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; has-vendor-device: 0; feature-xs-batcmd: 1; feature-wuRebootRequired: 1; feature-updatetime: 1; feature-ts2: 1; feature-ts: 1; feature-suspend: 1; feature-static-ip-setting: 1; feature-shutdown: 1; feature-setcomputername: 1; feature-s4: 1; feature-s3: 1; feature-reboot: 1; feature-poweroff: 1; feature-laptop-slate-mode: 1; feature-balloon: 1; data-ts: 0 live ( RO): true guest-metrics-last-updated ( RO): 20240403T22:07:14Z can-use-hotplug-vbd ( RO): true can-use-hotplug-vif ( RO): true cooperative ( RO) [DEPRECATED]: true tags (SRW): appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 5959171903355788617:4718727860986714682 hardware-platform-version ( RO): 0 has-vendor-device ( RW): false requires-reboot ( RO): false reference-label ( RO): windows-server-2022-64bit bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d -
Ok I think it's because the template used during the import is "Other install template" and that can cause some issues, eg on some specific VM settings.
Creating a new "empty" VM with a Windows 2022 template, shutdown and attach the disks from the imported VM and test again. You should see a similar performance. If it does, then we'll improve the import script to match the best template possible if we can (otherwise we'll let the user choose)
-
Created a new 2022 VM, same CPU/Memory. Deleted the VDI i made for that and added in the VM import VDI. Looks like its the same as the one above. It gets close but still a ~50MB/s 10k IOPS delta for the 4k random. Sequential is the exact same as the ISO installed VM. Much better than the initial import but assuming it's just the price to pay for a non-fresh install?
-
It's hard to tell since I'm not a Windows expert. Do you have some VMware guest tools or devices installed in there?
-
@olivierlambert VMWare Tools were removed on first boot and replaced with XenServer Tools 9.3.2. I also went into device manager just to be sure and removed anything with VMW too. It's close enough that it may work.
I appreciate your help! The template thing was 100% game changing for this situation.
-
Thanks for the feedback, I'll discuss the improvement with @florent
