Disaster Recovery backup auto-starting VMs (but shouldn't)
-
@olivierlambert Here ya go:
[07:47 VULCAN ~]# xe vm-param-list uuid=0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02 uuid ( RO) : 0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02 name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210718T093033Z) name-description ( RW): Windows Server 2016 License Manager & Print Server user-version ( RW): 2 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): 8589934592 memory-target ( RO): 0 memory-overhead ( RO): 73400320 memory-static-max ( RW): 8589934592 memory-dynamic-max ( RW): 8589934592 memory-dynamic-min ( RW): 2147483648 memory-static-min ( RW): 1073741824 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): weight: 256 VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e domain-type ( RW): hvm current-domain-type ( RO): unspecified HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: ndc HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): import_task: OpaqueRef:d0c0d5fe-8037-4f4e-a042-81c582f1eada; mac_seed: e4c5e202-c4f5-2807-c12b-3258fa268e27; xo:backup:datetime: 20210718T09:30:33Z; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 19700101T00:00:00Z install-time ( RO): 19700101T00:00:00Z VCPUs-number ( RO): 0 VCPUs-utilisation (MRO): os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0 PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): 7d8d958b-d20c-eb47-5229-bc80a988e1ad; e8176121-0637-c55e-61b5-e4b57fef14be networks (MRO): 0/ip: 169.254.10.199; 0/ipv4/0: 169.254.10.199; 0/ipv6/0: fe80:0000:0000:0000:84ec:876c:b5e6:0ac7 PV-drivers-detected ( RO): true other (MRO): shutdown: poweroff; feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1 live ( RO): true guest-metrics-last-updated ( RO): 20210719T17:00:36Z can-use-hotplug-vbd ( RO): true can-use-hotplug-vif ( RO): true cooperative ( RO) [DEPRECATED]: true tags (SRW): Disaster Recovery appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 4856085469031164830:1232191562809696746 hardware-platform-version ( RO): 2 has-vendor-device ( RW): true requires-reboot ( RO): false reference-label ( RO): windows-server-2016-64bit bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d -
Oh okay, you are using disaster recovery and not continuous replication
Sorry I missed that, despite it's in the title 
So there's no blocked operations in DR, at least it seems.
-
@bazzacad The replicated VM has its start operation blocked which means that it cannot automatically power-on.
But, if you force started this replicated VM via XO, it removed the blockage and now the VM will automatically power-on.
-
I just confirmed with tests that Disaster Recovery jobs do indeed set the
blocked-operationsparameter for backed up VMs. As Julien said, did you ever force-start this VM? That would have removedstartfrom theblocked-operationsfield, which is why we don't see it in your param-list, and why it's now being allowed to autostart. -
@fohdeesha
Thanks for the help everyone. I'm 100% sure I did not force start this VM. In fact all of the DR VMs were running. I know my screen shot it only shows 1 running, but I killed the others ASAP before taking the screen shot.
What I think happened is the secondary host crashed, so I rebooted it.
And I see this parameter is set: actions-after-crash ( RW): Restart.
Is it possible that when the host recovers from a crash it sends the signal to the VMs to force restart? I sent you the parameter of the DR VM that did start up date 2021-07-18 (above), but if I list the parameter from the DR VM for the next day 2021-07-19 the block parameter is set:[07:48 VULCAN ~]# xe vm-param-list uuid=e327b268-887d-395a-9665-2509c341df21 uuid ( RO) : e327b268-887d-395a-9665-2509c341df21 name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210719T165550Z) name-description ( RW): Windows Server 2016 License Manager & Print Server user-version ( RW): 2 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): 0 memory-target ( RO): 0 memory-overhead ( RO): 73400320 memory-static-max ( RW): 8589934592 memory-dynamic-max ( RW): 8589934592 memory-dynamic-min ( RW): 2147483648 memory-static-min ( RW): 1073741824 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): weight: 256 VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot current-operations (SRO): blocked-operations (MRW): start: Start operation for this vm is blocked, clone it if you want to use it. allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e domain-type ( RW): hvm current-domain-type ( RO): unspecified HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: ndc HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): xo:backup:sr: bbdc9045-8262-19de-53f6-3f1f448a06c7; import_task: OpaqueRef:1106f8b8-3ade-47c6-981f-b8c7a2100f0c; mac_seed: 229f9501-ea5d-79f5-b2ba-dac4f972a558; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:datetime: 20210719T16:55:50Z; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 19700101T00:00:00Z install-time ( RO): 19700101T00:00:00Z VCPUs-number ( RO): 0 VCPUs-utilisation (MRO): os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0 PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): 1b916155-4bc6-55da-df30-e115d913fff8; 315b8d45-14d6-79bd-4582-9b7a4820d120 networks (MRO): 0/ip: 192.168.254.30; 0/ipv4/0: 192.168.254.30; 0/ipv6/0: fe80:0000:0000:0000:7c2e:a532:d498:8b65 PV-drivers-detected ( RO): true other (MRO): feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1 live ( RO): true guest-metrics-last-updated ( RO): 20210718T15:56:35Z can-use-hotplug-vbd ( RO): true can-use-hotplug-vif ( RO): true cooperative ( RO) [DEPRECATED]: true tags (SRW): Disaster Recovery appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 7231909492726054475:6711832078023216933 hardware-platform-version ( RO): 2 has-vendor-device ( RW): true requires-reboot ( RO): false reference-label ( RO): windows-server-2016-64bit bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d -
No, a crash wouldn't trigger that. The only way to have this happen is:
- to force start the VM in XO
- or remove the
startfromblocked_operationselsewhere, eg via axe
But the first one seems more plausible
action-after-crashdoesn't have a related meaning (it's just that if your VM crashes, it will be restarted by Xen automatically). It's not host related. -
@olivierlambert
Well I just rebooted my secondary host again via iDrac & these DR VMs started up again.
Note, it started 2 of the same one. And I'm 110% sure I didn't force start any of these, just rebooted the host.
-
And those two got any blocked operations in their record?
-
@olivierlambert
No, the 2 DR VMs that auto-started, don't have the blocked-operations set, the other DR VMs from other days do have it set. -
That's the issue. Why they don't have it is the problem/mystery.
Try to check if new replicated VMs got it. If not, I would check if you are correctly up to date.
-
B bogdantomasciuc referenced this topic on
