Disaster Recovery backup auto-starting VMs (but shouldn't)
-
@olivierlambert Here ya go:
[07:47 VULCAN ~]# xe vm-param-list uuid=0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02 uuid ( RO) : 0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02 name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210718T093033Z) name-description ( RW): Windows Server 2016 License Manager & Print Server user-version ( RW): 2 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): 8589934592 memory-target ( RO): 0 memory-overhead ( RO): 73400320 memory-static-max ( RW): 8589934592 memory-dynamic-max ( RW): 8589934592 memory-dynamic-min ( RW): 2147483648 memory-static-min ( RW): 1073741824 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): weight: 256 VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot current-operations (SRO): blocked-operations (MRW): allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e domain-type ( RW): hvm current-domain-type ( RO): unspecified HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: ndc HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): import_task: OpaqueRef:d0c0d5fe-8037-4f4e-a042-81c582f1eada; mac_seed: e4c5e202-c4f5-2807-c12b-3258fa268e27; xo:backup:datetime: 20210718T09:30:33Z; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 19700101T00:00:00Z install-time ( RO): 19700101T00:00:00Z VCPUs-number ( RO): 0 VCPUs-utilisation (MRO): os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0 PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): 7d8d958b-d20c-eb47-5229-bc80a988e1ad; e8176121-0637-c55e-61b5-e4b57fef14be networks (MRO): 0/ip: 169.254.10.199; 0/ipv4/0: 169.254.10.199; 0/ipv6/0: fe80:0000:0000:0000:84ec:876c:b5e6:0ac7 PV-drivers-detected ( RO): true other (MRO): shutdown: poweroff; feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1 live ( RO): true guest-metrics-last-updated ( RO): 20210719T17:00:36Z can-use-hotplug-vbd ( RO): true can-use-hotplug-vif ( RO): true cooperative ( RO) [DEPRECATED]: true tags (SRW): Disaster Recovery appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 4856085469031164830:1232191562809696746 hardware-platform-version ( RO): 2 has-vendor-device ( RW): true requires-reboot ( RO): false reference-label ( RO): windows-server-2016-64bit bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d -
Oh okay, you are using disaster recovery and not continuous replication
Sorry I missed that, despite it's in the title 
So there's no blocked operations in DR, at least it seems.
-
@bazzacad The replicated VM has its start operation blocked which means that it cannot automatically power-on.
But, if you force started this replicated VM via XO, it removed the blockage and now the VM will automatically power-on.
-
I just confirmed with tests that Disaster Recovery jobs do indeed set the
blocked-operationsparameter for backed up VMs. As Julien said, did you ever force-start this VM? That would have removedstartfrom theblocked-operationsfield, which is why we don't see it in your param-list, and why it's now being allowed to autostart. -
@fohdeesha
Thanks for the help everyone. I'm 100% sure I did not force start this VM. In fact all of the DR VMs were running. I know my screen shot it only shows 1 running, but I killed the others ASAP before taking the screen shot.
What I think happened is the secondary host crashed, so I rebooted it.
And I see this parameter is set: actions-after-crash ( RW): Restart.
Is it possible that when the host recovers from a crash it sends the signal to the VMs to force restart? I sent you the parameter of the DR VM that did start up date 2021-07-18 (above), but if I list the parameter from the DR VM for the next day 2021-07-19 the block parameter is set:[07:48 VULCAN ~]# xe vm-param-list uuid=e327b268-887d-395a-9665-2509c341df21 uuid ( RO) : e327b268-887d-395a-9665-2509c341df21 name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210719T165550Z) name-description ( RW): Windows Server 2016 License Manager & Print Server user-version ( RW): 2 is-a-template ( RW): false is-default-template ( RW): false is-a-snapshot ( RO): false snapshot-of ( RO): <not in database> snapshots ( RO): snapshot-time ( RO): 19700101T00:00:00Z snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running parent ( RO): <not in database> children ( RO): is-control-domain ( RO): false power-state ( RO): halted memory-actual ( RO): 0 memory-target ( RO): 0 memory-overhead ( RO): 73400320 memory-static-max ( RW): 8589934592 memory-dynamic-max ( RW): 8589934592 memory-dynamic-min ( RW): 2147483648 memory-static-min ( RW): 1073741824 suspend-VDI-uuid ( RW): <not in database> suspend-SR-uuid ( RW): <not in database> VCPUs-params (MRW): weight: 256 VCPUs-max ( RW): 4 VCPUs-at-startup ( RW): 4 actions-after-shutdown ( RW): Destroy actions-after-reboot ( RW): Restart actions-after-crash ( RW): Restart console-uuids (SRO): hvm ( RO): false platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot current-operations (SRO): blocked-operations (MRW): start: Start operation for this vm is blocked, clone it if you want to use it. allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6 possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e domain-type ( RW): hvm current-domain-type ( RO): unspecified HVM-boot-policy ( RW): BIOS order HVM-boot-params (MRW): order: ndc HVM-shadow-multiplier ( RW): 1.000 PV-kernel ( RW): PV-ramdisk ( RW): PV-args ( RW): PV-legacy-args ( RW): PV-bootloader ( RW): PV-bootloader-args ( RW): last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000 last-boot-record ( RO): '' resident-on ( RO): <not in database> affinity ( RW): <not in database> other-config (MRW): xo:backup:sr: bbdc9045-8262-19de-53f6-3f1f448a06c7; import_task: OpaqueRef:1106f8b8-3ade-47c6-981f-b8c7a2100f0c; mac_seed: 229f9501-ea5d-79f5-b2ba-dac4f972a558; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:datetime: 20210719T16:55:50Z; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom dom-id ( RO): -1 recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions> xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data: ha-always-run ( RW) [DEPRECATED]: false ha-restart-priority ( RW): blobs ( RO): start-time ( RO): 19700101T00:00:00Z install-time ( RO): 19700101T00:00:00Z VCPUs-number ( RO): 0 VCPUs-utilisation (MRO): os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0 PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42 PV-drivers-up-to-date ( RO) [DEPRECATED]: true memory (MRO): disks (MRO): VBDs (SRO): 1b916155-4bc6-55da-df30-e115d913fff8; 315b8d45-14d6-79bd-4582-9b7a4820d120 networks (MRO): 0/ip: 192.168.254.30; 0/ipv4/0: 192.168.254.30; 0/ipv6/0: fe80:0000:0000:0000:7c2e:a532:d498:8b65 PV-drivers-detected ( RO): true other (MRO): feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1 live ( RO): true guest-metrics-last-updated ( RO): 20210718T15:56:35Z can-use-hotplug-vbd ( RO): true can-use-hotplug-vif ( RO): true cooperative ( RO) [DEPRECATED]: true tags (SRW): Disaster Recovery appliance ( RW): <not in database> snapshot-schedule ( RW): <not in database> is-vmss-snapshot ( RO): false start-delay ( RW): 0 shutdown-delay ( RW): 0 order ( RW): 0 version ( RO): 0 generation-id ( RO): 7231909492726054475:6711832078023216933 hardware-platform-version ( RO): 2 has-vendor-device ( RW): true requires-reboot ( RO): false reference-label ( RO): windows-server-2016-64bit bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d -
No, a crash wouldn't trigger that. The only way to have this happen is:
- to force start the VM in XO
- or remove the
startfromblocked_operationselsewhere, eg via axe
But the first one seems more plausible
action-after-crashdoesn't have a related meaning (it's just that if your VM crashes, it will be restarted by Xen automatically). It's not host related. -
@olivierlambert
Well I just rebooted my secondary host again via iDrac & these DR VMs started up again.
Note, it started 2 of the same one. And I'm 110% sure I didn't force start any of these, just rebooted the host.
-
And those two got any blocked operations in their record?
-
@olivierlambert
No, the 2 DR VMs that auto-started, don't have the blocked-operations set, the other DR VMs from other days do have it set. -
That's the issue. Why they don't have it is the problem/mystery.
Try to check if new replicated VMs got it. If not, I would check if you are correctly up to date.
-
B bogdantomasciuc referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login