Migrating VM fails with DUPLICATE_VM error part2

AndreS

[20:10 xen2 ~]# xe vm-migrate remote-master=172.25.10.11 remote-username=root remote-password=xxxxxxxx vif:f4b175c2-0082-212c-b9d9-bd616cd83d2c=a014b230-2db6-adb4-ba4f-0b1cc07fdcae vm=Win10vm
Performing a Storage XenMotion migration. Your VM's VDIs will be migrated with the VM.
Will migrate to remote host: xen1, using remote network: Pool-wide network associated with eth0. Here is the VDI mapping:
VDI 4d4a809d-6801-4462-8e52-811882106821 -> SR 270f8f4a-a24c-ced6-99c7-9bc2ba5f5008
VDI 6a30ca10-a386-4e00-91aa-89c3e5bd43de -> SR 270f8f4a-a24c-ced6-99c7-9bc2ba5f5008
Cannot restore this VM because it would create a duplicate
vm: abd338e4-0ae1-24fa-38be-91fc0fe57332

olivierlambert

That's weird indeed. Maybe it's not the VM itself that will be duplicated, but an object with it.

Have you checked MAC address on destination? There's something we need to dig in XAPI to learn what it means in terms of duplicate

olivierlambert

Yeah after taking a look, found this: https://github.com/xapi-project/xen-api/blob/d496b90c4172f71337841dcacb3496751141f712/ocaml/xapi/import.ml#L246

It might be duplicated MAC address related.

AndreS

@olivierlambert
Funny you say that, look at where we both commented on last year
https://xcp-ng.org/forum/topic/3182/centos-ovas-imported-from-esx-network-hang/34?_=1636230593687

Checked the MAC addresses of the two vms (the imported ones) but they are different. I'll try a side by side comparison of all the objects.

AndreS

@andres
Both vms exported from esxi

Windows 10 vm

other-config (MRW): auto_poweron: true; import_task: OpaqueRef:680ed8f5-22a5-4a4b-af3b-6988f7734441; install-repository: cdrom; vgpu_pci: ; base_template_name: Other install media; mac_seed: 5e88eb6a-d680-c47f-a94a-028886971ba4; install-methods: cdrom

Gentoo linux vm

other-config (MRW): auto_poweron: true; vgpu_pci: ; base_template_name: Other install media; mac_seed: 5e88eb6a-d680-c47f-a94a-028886971ba4; install-methods: cdrom

the mac-seed id is the same.

olivierlambert

Can you check if you don't have any similar MAC on the destination host?

About the seed, well, in theory that shouldn't be a problem, but you can always replace it by another random UUID (here is one I got with uuidgen: 3ec7fc14-1e25-4453-b3f9-5a7b6554d241).

AndreS

@olivierlambert
Similar, but not the same; the two imported vms have MAC addresses that differ only by one ( 9e:86:37:32:02:d7 vs 9e:86:37:32:02:d8) which should be enough.
Visually checked all MAC addresses on the destination host via XO, nothing even close except the D7 vs D8.
Is there a way to dump all object from all vms on a host?

olivierlambert

Yes but I don't remember. Let me try if I can find this quickly.

olivierlambert

You can do a xe vm-vif-list params=vm-name-label,MAC to see all your VMs MACs at once.

AndreS

@olivierlambert Thanks, this confirms that there is no MAC address clash.
The fact that they are both imported the same way from the same source system is a strong suggestion that something is clashing, especially after the MAC address issue found last year.

Suggestions are welcome, otherwise I might entertain myself and recreate a Windows 10 vm on the destination and symply copy over and connect the disks.

Let me know if you want me to provide details or create a ticket somewhere.

olivierlambert

Well, I'd love that we could find the culprit

xe vm-list params=all will give you all the params for all VMs on a given host.

Then, you should try to compare between 2 hosts and see if there's similar fields that shouldn't be similar (I know, it's vague).

In the mean time, let me ask around to some XAPI devs.

AndreS

@olivierlambert Appreciated. I will go through the fields to see if anything looks odd..

AndreS

@olivierlambert Nothing odd that stands out for me when comparing the data.

Additionally I tried migrating the (most likely) offending vm from the other host and I get the same error.
Migrated two other (created on xcp-ng) vms but those fail with a VDI_COPY_FAILED but the migration starts properly. Only at the end I get this error.

vm.migrate
{
  "vm": "0ab24395-becb-71f4-4f93-066c2d660cff",
  "mapVifsNetworks": {
    "651db47e-129a-7152-f988-cc4408abce0a": "a014b230-2db6-adb4-ba4f-0b1cc07fdcae"
  },
  "migrationNetwork": "a014b230-2db6-adb4-ba4f-0b1cc07fdcae",
  "sr": "270f8f4a-a24c-ced6-99c7-9bc2ba5f5008",
  "targetHost": "3b57d90b-983f-46bb-8f52-4319025d1182"
}
{
  "code": 21,
  "data": {
    "objectId": "0ab24395-becb-71f4-4f93-066c2d660cff",
    "code": "VDI_COPY_FAILED"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at operationFailed (/opt/xo/xo-builds/xen-orchestra-202111061645/packages/xo-common/src/api-errors.js:21:32)
    at file:///opt/xo/xo-builds/xen-orchestra-202111061645/packages/xo-server/src/api/vm.mjs:482:15
    at Object.migrate (file:///opt/xo/xo-builds/xen-orchestra-202111061645/packages/xo-server/src/api/vm.mjs:469:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202111061645/packages/xo-server/src/xo-mixins/api.mjs:304:20)"
}

Now I need to figure out what the real issue is.

olivierlambert

Can you also try with xe too? Maybe the error message will be more clear.

AndreS

@olivierlambert Life happened, just spent some time working through scenarios.

Using xe vm-migrate I can move vms from one host to the other (this is across pools) and back. For me this rules out a host level issue.

When I try to move the vms that won't migrate in both cases I get the error

Cannot restore this VM because it would create a duplicate

This clearly points at the two vms that were imported from esxi last year.

Reviewed the output of xe vm-list params=all uuid=<relevant vm uuid> again and finaly noticed that they both have the same mac_seed: value under other-config (MRW):

other-config (MRW): auto_poweron: true; vgpu_pci: ; base_template_name: Other install media; mac_seed: 5e88eb6a-d680-c47f-a94a-028886971ba4; install-methods: cdrom

and

other-config (MRW): auto_poweron: true; import_task: OpaqueRef:680ed8f5-22a5-4a4b-af3b-6988f7734441; install-repository: cdrom; vgpu_pci: ; base_template_name: Other install media; mac_seed: 5e88eb6a-d680-c47f-a94a-028886971ba4; install-methods: cdrom

Could that cause a clash/duplicate error?

olivierlambert

Please modify the mac seed and see if it still trigger the error

AndreS

@olivierlambert Thanks. Done that.
Next stop is

xe vm-migrate remote-master=172.25.10.11 remote-username=root remote-password=xxxxxx vif:f4b175c2-0082-212c-b9d9-bd616cd83d2c=a014b230-2db6-adb4-ba4f-0b1cc07fdcae vm=Win10vm
Performing a Storage XenMotion migration. Your VM's VDIs will be migrated with the VM.
Will migrate to remote host: xen1, using remote network: Pool-wide network associated with eth0. Here is the VDI mapping:
VDI 4d4a809d-6801-4462-8e52-811882106821 -> SR 270f8f4a-a24c-ced6-99c7-9bc2ba5f5008
VDI 6a30ca10-a386-4e00-91aa-89c3e5bd43de -> SR 270f8f4a-a24c-ced6-99c7-9bc2ba5f5008
There were no servers available to complete the specified operation.

Doesn't matter what I set the mac_seed to: empty, just a few characters or change one character. Setting it back to the 'original' value reproduces the earlier error about creating a duplicate so the mac_seed value is making a difference. But there seems to be more.

I couldn't find anything specific on There were no servers available to complete the specified operation. on the forum, just one hit; someone asking what it means.

A wider internet search showed some Citrix /Xen related posts; one about a DVD image not being available anymore so I disabled the DVD but that also made no difference. There is more than enough storage available on the receiving SR; more than double what is needed.

But now it gets interesting. When I try to start the vm again I get this:

vm.start
{
  "id": "afe623be-5451-fd48-3f24-60120e53f5ab",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "code": "NO_HOSTS_AVAILABLE",
  "params": [],
  "call": {
    "method": "VM.start",
    "params": [
      "OpaqueRef:3947721f-7307-4560-aa59-dec8a8e26bfb",
      false,
      false
    ]
  },
  "message": "NO_HOSTS_AVAILABLE()",
  "name": "XapiError",
  "stack": "XapiError: NO_HOSTS_AVAILABLE()
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202111061645/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202111061645/packages/xen-api/src/transports/json-rpc.js:41:27
    at AsyncResource.runInAsyncScope (async_hooks.js:197:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202111061645/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (internal/timers.js:464:21)
    at process.topLevelDomainCallback (domain.js:152:15)
    at process.callbackTrampoline (internal/async_hooks.js:128:24)"
}

Setting it back to the 'original' mac_seed value does not even allow me to start the vm. I do get the ' duplicate' error on vm migration.

When I start via xe I get

xe vm-start vm=Win10vm
There are no suitable hosts to start this VM on.
The following table provides per-host reasons for why the VM could not be started:

xen2    : Cannot start here [Not enough free memory]


There were no servers available to complete the specified operation.

There is enough free memory (9 Gb available, 6 Gb requested). Even freeing up 15 Gb of memory does not make a diffence. Same errors in XOA and via xe.

Any other suggestions?

olivierlambert

So it seems that MAC SEED was the initial issue?
Can you display the xe vm-param-list uuid=<VM UUID>?

AndreS

@olivierlambert Here you go

                         snapshot-time ( RO): 19700101T00:00:00Z
                         snapshot-info ( RO): 
                                parent ( RO): <not in database>
                              children ( RO): 
                     is-control-domain ( RO): false
                           power-state ( RO): halted
                         memory-actual ( RO): 6442455040
                         memory-target ( RO): 0
                       memory-overhead ( RO): 54525952
                     memory-static-max ( RW): 6442450944
                    memory-dynamic-max ( RW): 6442450944
                    memory-dynamic-min ( RW): 6442450944
                     memory-static-min ( RW): 6442450944
                      suspend-VDI-uuid ( RW): <not in database>
                       suspend-SR-uuid ( RW): <not in database>
                          VCPUs-params (MRW): 
                             VCPUs-max ( RW): 2
                      VCPUs-at-startup ( RW): 2
                actions-after-shutdown ( RW): Destroy
                  actions-after-reboot ( RW): Restart
                   actions-after-crash ( RW): Restart
                         console-uuids (SRO): 
                                   hvm ( RO): false
                              platform (MRW): timeoffset: 3570; device-model: qemu-upstream-compat; nx: true; acpi: 1; apic: true; pae: true; hpet: true; viridian: true
                    allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot
                    current-operations (SRO): 
                    blocked-operations (MRW): 
                   allowed-VBD-devices (SRO): 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                   allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
                        possible-hosts ( RO): 
                           domain-type ( RW): hvm
                   current-domain-type ( RO): unspecified
                       HVM-boot-policy ( RW): BIOS order
                       HVM-boot-params (MRW): order: cnd
                 HVM-shadow-multiplier ( RW): 1.000
                             PV-kernel ( RW): 
                            PV-ramdisk ( RW): 
                               PV-args ( RW): 
                        PV-legacy-args ( RW): 
                         PV-bootloader ( RW): 
                    PV-bootloader-args ( RW): 
                   last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-f7fa3223-2d93fbff-00000423-00000001-000007ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                      last-boot-record ( RO): '{"xen_platform":[1,1],"pv_drivers_detected":true,"pci_power_mgmt":false,"pci_msitranslate":true,"qemu_vifs":[],"qemu_vbds":[],"suspend_memory_bytes":8589934592,"original_profile":"Qemu_upstream_compat","profile":"Qemu_upstream_compat","nested_virt":false,"nomigrate":false,"domain_config":["X86",{"emulation_flags":["X86_EMU_LAPIC","X86_EMU_HPET","X86_EMU_PM","X86_EMU_RTC","X86_EMU_IOAPIC","X86_EMU_PIC","X86_EMU_VGA","X86_EMU_IOMMU","X86_EMU_PIT","X86_EMU_USE_PIRQ"]}],"last_start_time":1603780055.119536,"ty":["HVM",{"firmware":"Bios","qemu_stubdom":false,"qemu_disk_cmdline":false,"boot_order":"cdn","pci_passthrough":false,"pci_emulations":[],"serial":"pty","acpi":true,"video":"Cirrus","video_mib":4,"timeoffset":"3585","shadow_multiplier":1.0,"hap":true}],"build_info":{"has_hard_affinity":false,"priv":["BuildHVM",{"video_mib":4,"shadow_multiplier":1.0}],"vcpus":2,"kernel":"/usr/libexec/xen/boot/hvmloader","memory_target":8388608,"memory_max":8388608},"version":2}'
                           resident-on ( RO): <not in database>
                              affinity ( RW): <not in database>
                          other-config (MRW): mac_seed: 5e88eb6a-d680-c47f-a94a-028886971ba4; auto_poweron: true; vgpu_pci: ; base_template_name: Other install media; install-methods: cdrom
                                dom-id ( RO): -1
                       recommendations ( RO): <restrictions><restriction field="memory-static-max" max="137438953472" /><restriction field="vcpus-max" max="32" /><restriction property="number-of-vbds" max="255" /><restriction property="number-of-vifs" max="7" /><restriction field="has-vendor-device" value="false" /></restrictions>
                         xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: 
            ha-always-run ( RW) [DEPRECATED]: false
                   ha-restart-priority ( RW): 
                                 blobs ( RO): 
                            start-time ( RO): 19700101T00:00:00Z
                          install-time ( RO): 19700101T00:00:00Z
                          VCPUs-number ( RO): 0
                     VCPUs-utilisation (MRO): 
                            os-version (MRO): name: Microsoft Windows 10 Pro|C:\WINDOWS|\Device\Harddisk0\Partition2; distro: windows; major: 6; minor: 2; spmajor: 0; spminor: 0
                    PV-drivers-version (MRO): major: 0; minor: 0; micro: 0; build: 0
    PV-drivers-up-to-date ( RO) [DEPRECATED]: true
                                memory (MRO): 
                                 disks (MRO): 
                                  VBDs (SRO): ac9e1b28-f576-b83d-c7ca-8ce097e3b30e; 9177269c-442b-24be-174e-25ba7ba20846; 3adc01b3-e9a7-42d8-8375-7380ef9c2791
                              networks (MRO): 0/ip: 172.25.10.211; 0/ipv4/0: 172.25.10.211
                   PV-drivers-detected ( RO): true
                                 other (MRO): feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: CreateProcessAsUser : 2 failed.; feature-s4: 1; feature-s3: 1; feature-reboot: 1; feature-poweroff: 1; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 0; platform-feature-xs_reset_watches: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1
                                  live ( RO): true
            guest-metrics-last-updated ( RO): 20211114T22:16:45Z
                   can-use-hotplug-vbd ( RO): unspecified
                   can-use-hotplug-vif ( RO): true
              cooperative ( RO) [DEPRECATED]: true
                                  tags (SRW): 
                             appliance ( RW): <not in database>
                     snapshot-schedule ( RW): <not in database>
                      is-vmss-snapshot ( RO): false
                           start-delay ( RW): 0
                        shutdown-delay ( RW): 0
                                 order ( RW): 0
                               version ( RO): 0
                         generation-id ( RO): 2464683622889442979:3979947479560168523
             hardware-platform-version ( RO): 0
                     has-vendor-device ( RW): false
                       requires-reboot ( RO): false
                       reference-label ( RO): 
                          bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d

olivierlambert

Okay and on the host in question, can you put a xe host-param-list?