XCP-ng

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups

    Disaster Recovery backup auto-starting VMs (but shouldn't)

    Xen Orchestra
    4
    15
    555
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • julien-f
      julien-f Vates 🪐 Co-Founder🦸 XO Team 🔭 @olivierlambert last edited by

      That should not be the case as the replicated VMs have their start operation blocked 🤔

      What version of XCP-ng are running on this secondary host?

      B 1 Reply Last reply Reply Quote 0
      • B
        bazzacad @julien-f last edited by

        @julien-f
        8.2.0

        652fd1e8-e0df-4756-b6db-d9d6b558fae3-image.png

        julien-f 1 Reply Last reply Reply Quote 0
        • olivierlambert
          olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

          Can you do a xe vm-param-list uuid=<UUID> on any on the replicated VM that is auto-booting? (the VM on destination, the copy)

          B 1 Reply Last reply Reply Quote 0
          • B
            bazzacad @olivierlambert last edited by bazzacad

            @olivierlambert Here ya go:

            [07:47 VULCAN ~]# xe vm-param-list uuid=0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02
            uuid ( RO)                                  : 0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02
                                        name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210718T093033Z)
                                  name-description ( RW): Windows Server 2016 License Manager & Print Server
                                      user-version ( RW): 2
                                     is-a-template ( RW): false
                               is-default-template ( RW): false
                                     is-a-snapshot ( RO): false
                                       snapshot-of ( RO): <not in database>
                                         snapshots ( RO):
                                     snapshot-time ( RO): 19700101T00:00:00Z
                                     snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running
                                            parent ( RO): <not in database>
                                          children ( RO):
                                 is-control-domain ( RO): false
                                       power-state ( RO): halted
                                     memory-actual ( RO): 8589934592
                                     memory-target ( RO): 0
                                   memory-overhead ( RO): 73400320
                                 memory-static-max ( RW): 8589934592
                                memory-dynamic-max ( RW): 8589934592
                                memory-dynamic-min ( RW): 2147483648
                                 memory-static-min ( RW): 1073741824
                                  suspend-VDI-uuid ( RW): <not in database>
                                   suspend-SR-uuid ( RW): <not in database>
                                      VCPUs-params (MRW): weight: 256
                                         VCPUs-max ( RW): 4
                                  VCPUs-at-startup ( RW): 4
                            actions-after-shutdown ( RW): Destroy
                              actions-after-reboot ( RW): Restart
                               actions-after-crash ( RW): Restart
                                     console-uuids (SRO):
                                               hvm ( RO): false
                                          platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true
                                allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot
                                current-operations (SRO):
                                blocked-operations (MRW):
                               allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                               allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
                                    possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e
                                       domain-type ( RW): hvm
                               current-domain-type ( RO): unspecified
                                   HVM-boot-policy ( RW): BIOS order
                                   HVM-boot-params (MRW): order: ndc
                             HVM-shadow-multiplier ( RW): 1.000
                                         PV-kernel ( RW):
                                        PV-ramdisk ( RW):
                                           PV-args ( RW):
                                    PV-legacy-args ( RW):
                                     PV-bootloader ( RW):
                                PV-bootloader-args ( RW):
                               last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                                  last-boot-record ( RO): ''
                                       resident-on ( RO): <not in database>
                                          affinity ( RW): <not in database>
                                      other-config (MRW): import_task: OpaqueRef:d0c0d5fe-8037-4f4e-a042-81c582f1eada; mac_seed: e4c5e202-c4f5-2807-c12b-3258fa268e27; xo:backup:datetime: 20210718T09:30:33Z; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom
                                            dom-id ( RO): -1
                                   recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                                     xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data:
                        ha-always-run ( RW) [DEPRECATED]: false
                               ha-restart-priority ( RW):
                                             blobs ( RO):
                                        start-time ( RO): 19700101T00:00:00Z
                                      install-time ( RO): 19700101T00:00:00Z
                                      VCPUs-number ( RO): 0
                                 VCPUs-utilisation (MRO):
                                        os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0
                                PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42
                PV-drivers-up-to-date ( RO) [DEPRECATED]: true
                                            memory (MRO):
                                             disks (MRO):
                                              VBDs (SRO): 7d8d958b-d20c-eb47-5229-bc80a988e1ad; e8176121-0637-c55e-61b5-e4b57fef14be
                                          networks (MRO): 0/ip: 169.254.10.199; 0/ipv4/0: 169.254.10.199; 0/ipv6/0: fe80:0000:0000:0000:84ec:876c:b5e6:0ac7
                               PV-drivers-detected ( RO): true
                                             other (MRO): shutdown: poweroff; feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1
                                              live ( RO): true
                        guest-metrics-last-updated ( RO): 20210719T17:00:36Z
                               can-use-hotplug-vbd ( RO): true
                               can-use-hotplug-vif ( RO): true
                          cooperative ( RO) [DEPRECATED]: true
                                              tags (SRW): Disaster Recovery
                                         appliance ( RW): <not in database>
                                 snapshot-schedule ( RW): <not in database>
                                  is-vmss-snapshot ( RO): false
                                       start-delay ( RW): 0
                                    shutdown-delay ( RW): 0
                                             order ( RW): 0
                                           version ( RO): 0
                                     generation-id ( RO): 4856085469031164830:1232191562809696746
                         hardware-platform-version ( RO): 2
                                 has-vendor-device ( RW): true
                                   requires-reboot ( RO): false
                                   reference-label ( RO): windows-server-2016-64bit
                                      bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d
            
            
            fohdeesha 1 Reply Last reply Reply Quote 0
            • olivierlambert
              olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

              Oh okay, you are using disaster recovery and not continuous replication 🙂 Sorry I missed that, despite it's in the title 😛

              So there's no blocked operations in DR, at least it seems.

              1 Reply Last reply Reply Quote 0
              • julien-f
                julien-f Vates 🪐 Co-Founder🦸 XO Team 🔭 @bazzacad last edited by

                @bazzacad The replicated VM has its start operation blocked which means that it cannot automatically power-on.

                But, if you force started this replicated VM via XO, it removed the blockage and now the VM will automatically power-on.

                1 Reply Last reply Reply Quote 0
                • fohdeesha
                  fohdeesha Vates 🪐 Pro Support Team 💡 @bazzacad last edited by

                  I just confirmed with tests that Disaster Recovery jobs do indeed set the blocked-operations parameter for backed up VMs. As Julien said, did you ever force-start this VM? That would have removed start from the blocked-operations field, which is why we don't see it in your param-list, and why it's now being allowed to autostart.

                  B 1 Reply Last reply Reply Quote 0
                  • B
                    bazzacad @fohdeesha last edited by

                    @fohdeesha
                    Thanks for the help everyone. I'm 100% sure I did not force start this VM. In fact all of the DR VMs were running. I know my screen shot it only shows 1 running, but I killed the others ASAP before taking the screen shot.
                    What I think happened is the secondary host crashed, so I rebooted it.
                    And I see this parameter is set: actions-after-crash ( RW): Restart.
                    Is it possible that when the host recovers from a crash it sends the signal to the VMs to force restart? I sent you the parameter of the DR VM that did start up date 2021-07-18 (above), but if I list the parameter from the DR VM for the next day 2021-07-19 the block parameter is set:

                    [07:48 VULCAN ~]# xe vm-param-list uuid=e327b268-887d-395a-9665-2509c341df21
                    uuid ( RO)                                  : e327b268-887d-395a-9665-2509c341df21
                                                name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210719T165550Z)
                                          name-description ( RW): Windows Server 2016 License Manager & Print Server
                                              user-version ( RW): 2
                                             is-a-template ( RW): false
                                       is-default-template ( RW): false
                                             is-a-snapshot ( RO): false
                                               snapshot-of ( RO): <not in database>
                                                 snapshots ( RO):
                                             snapshot-time ( RO): 19700101T00:00:00Z
                                             snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running
                                                    parent ( RO): <not in database>
                                                  children ( RO):
                                         is-control-domain ( RO): false
                                               power-state ( RO): halted
                                             memory-actual ( RO): 0
                                             memory-target ( RO): 0
                                           memory-overhead ( RO): 73400320
                                         memory-static-max ( RW): 8589934592
                                        memory-dynamic-max ( RW): 8589934592
                                        memory-dynamic-min ( RW): 2147483648
                                         memory-static-min ( RW): 1073741824
                                          suspend-VDI-uuid ( RW): <not in database>
                                           suspend-SR-uuid ( RW): <not in database>
                                              VCPUs-params (MRW): weight: 256
                                                 VCPUs-max ( RW): 4
                                          VCPUs-at-startup ( RW): 4
                                    actions-after-shutdown ( RW): Destroy
                                      actions-after-reboot ( RW): Restart
                                       actions-after-crash ( RW): Restart
                                             console-uuids (SRO):
                                                       hvm ( RO): false
                                                  platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true
                                        allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot
                                        current-operations (SRO):
                                        blocked-operations (MRW): start: Start operation for this vm is blocked, clone it if you want to use it.
                                       allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                                       allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
                                            possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e
                                               domain-type ( RW): hvm
                                       current-domain-type ( RO): unspecified
                                           HVM-boot-policy ( RW): BIOS order
                                           HVM-boot-params (MRW): order: ndc
                                     HVM-shadow-multiplier ( RW): 1.000
                                                 PV-kernel ( RW):
                                                PV-ramdisk ( RW):
                                                   PV-args ( RW):
                                            PV-legacy-args ( RW):
                                             PV-bootloader ( RW):
                                        PV-bootloader-args ( RW):
                                       last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                                          last-boot-record ( RO): ''
                                               resident-on ( RO): <not in database>
                                                  affinity ( RW): <not in database>
                                              other-config (MRW): xo:backup:sr: bbdc9045-8262-19de-53f6-3f1f448a06c7; import_task: OpaqueRef:1106f8b8-3ade-47c6-981f-b8c7a2100f0c; mac_seed: 229f9501-ea5d-79f5-b2ba-dac4f972a558; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:datetime: 20210719T16:55:50Z; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom
                                                    dom-id ( RO): -1
                                           recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                                             xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data:
                                ha-always-run ( RW) [DEPRECATED]: false
                                       ha-restart-priority ( RW):
                                                     blobs ( RO):
                                                start-time ( RO): 19700101T00:00:00Z
                                              install-time ( RO): 19700101T00:00:00Z
                                              VCPUs-number ( RO): 0
                                         VCPUs-utilisation (MRO):
                                                os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0
                                        PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42
                        PV-drivers-up-to-date ( RO) [DEPRECATED]: true
                                                    memory (MRO):
                                                     disks (MRO):
                                                      VBDs (SRO): 1b916155-4bc6-55da-df30-e115d913fff8; 315b8d45-14d6-79bd-4582-9b7a4820d120
                                                  networks (MRO): 0/ip: 192.168.254.30; 0/ipv4/0: 192.168.254.30; 0/ipv6/0: fe80:0000:0000:0000:7c2e:a532:d498:8b65
                                       PV-drivers-detected ( RO): true
                                                     other (MRO): feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1
                                                      live ( RO): true
                                guest-metrics-last-updated ( RO): 20210718T15:56:35Z
                                       can-use-hotplug-vbd ( RO): true
                                       can-use-hotplug-vif ( RO): true
                                  cooperative ( RO) [DEPRECATED]: true
                                                      tags (SRW): Disaster Recovery
                                                 appliance ( RW): <not in database>
                                         snapshot-schedule ( RW): <not in database>
                                          is-vmss-snapshot ( RO): false
                                               start-delay ( RW): 0
                                            shutdown-delay ( RW): 0
                                                     order ( RW): 0
                                                   version ( RO): 0
                                             generation-id ( RO): 7231909492726054475:6711832078023216933
                                 hardware-platform-version ( RO): 2
                                         has-vendor-device ( RW): true
                                           requires-reboot ( RO): false
                                           reference-label ( RO): windows-server-2016-64bit
                                              bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d
                    
                    1 Reply Last reply Reply Quote 0
                    • olivierlambert
                      olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by olivierlambert

                      No, a crash wouldn't trigger that. The only way to have this happen is:

                      • to force start the VM in XO
                      • or remove the start from blocked_operations elsewhere, eg via a xe

                      But the first one seems more plausible 🙂

                      action-after-crash doesn't have a related meaning (it's just that if your VM crashes, it will be restarted by Xen automatically). It's not host related.

                      B 1 Reply Last reply Reply Quote 0
                      • B
                        bazzacad @olivierlambert last edited by

                        @olivierlambert
                        Well I just rebooted my secondary host again via iDrac & these DR VMs started up again.
                        Note, it started 2 of the same one. And I'm 110% sure I didn't force start any of these, just rebooted the host.

                        248a113a-69f7-4804-a6dd-eff91e281184-image.png

                        1 Reply Last reply Reply Quote 0
                        • olivierlambert
                          olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

                          And those two got any blocked operations in their record?

                          B 1 Reply Last reply Reply Quote 0
                          • B
                            bazzacad @olivierlambert last edited by

                            @olivierlambert
                            No, the 2 DR VMs that auto-started, don't have the blocked-operations set, the other DR VMs from other days do have it set.

                            1 Reply Last reply Reply Quote 0
                            • olivierlambert
                              olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

                              That's the issue. Why they don't have it is the problem/mystery.

                              Try to check if new replicated VMs got it. If not, I would check if you are correctly up to date.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post