XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Disaster Recovery backup auto-starting VMs (but shouldn't)

    Scheduled Pinned Locked Moved Xen Orchestra
    15 Posts 4 Posters 1.9k Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      Pinging @pdonias or @julien-f

      julien-fJ 1 Reply Last reply Reply Quote 0
      • julien-fJ Offline
        julien-f Vates 🪐 Co-Founder XO Team @olivierlambert
        last edited by

        That should not be the case as the replicated VMs have their start operation blocked 🤔

        What version of XCP-ng are running on this secondary host?

        B 1 Reply Last reply Reply Quote 0
        • B Offline
          bazzacad @julien-f
          last edited by

          @julien-f
          8.2.0

          652fd1e8-e0df-4756-b6db-d9d6b558fae3-image.png

          julien-fJ 1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Can you do a xe vm-param-list uuid=<UUID> on any on the replicated VM that is auto-booting? (the VM on destination, the copy)

            B 1 Reply Last reply Reply Quote 0
            • B Offline
              bazzacad @olivierlambert
              last edited by bazzacad

              @olivierlambert Here ya go:

              [07:47 VULCAN ~]# xe vm-param-list uuid=0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02
              uuid ( RO)                                  : 0b6d3bff-0c27-9b67-d1e6-dd6f0f702e02
                                          name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210718T093033Z)
                                    name-description ( RW): Windows Server 2016 License Manager & Print Server
                                        user-version ( RW): 2
                                       is-a-template ( RW): false
                                 is-default-template ( RW): false
                                       is-a-snapshot ( RO): false
                                         snapshot-of ( RO): <not in database>
                                           snapshots ( RO):
                                       snapshot-time ( RO): 19700101T00:00:00Z
                                       snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running
                                              parent ( RO): <not in database>
                                            children ( RO):
                                   is-control-domain ( RO): false
                                         power-state ( RO): halted
                                       memory-actual ( RO): 8589934592
                                       memory-target ( RO): 0
                                     memory-overhead ( RO): 73400320
                                   memory-static-max ( RW): 8589934592
                                  memory-dynamic-max ( RW): 8589934592
                                  memory-dynamic-min ( RW): 2147483648
                                   memory-static-min ( RW): 1073741824
                                    suspend-VDI-uuid ( RW): <not in database>
                                     suspend-SR-uuid ( RW): <not in database>
                                        VCPUs-params (MRW): weight: 256
                                           VCPUs-max ( RW): 4
                                    VCPUs-at-startup ( RW): 4
                              actions-after-shutdown ( RW): Destroy
                                actions-after-reboot ( RW): Restart
                                 actions-after-crash ( RW): Restart
                                       console-uuids (SRO):
                                                 hvm ( RO): false
                                            platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true
                                  allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot
                                  current-operations (SRO):
                                  blocked-operations (MRW):
                                 allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                                 allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
                                      possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e
                                         domain-type ( RW): hvm
                                 current-domain-type ( RO): unspecified
                                     HVM-boot-policy ( RW): BIOS order
                                     HVM-boot-params (MRW): order: ndc
                               HVM-shadow-multiplier ( RW): 1.000
                                           PV-kernel ( RW):
                                          PV-ramdisk ( RW):
                                             PV-args ( RW):
                                      PV-legacy-args ( RW):
                                       PV-bootloader ( RW):
                                  PV-bootloader-args ( RW):
                                 last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                                    last-boot-record ( RO): ''
                                         resident-on ( RO): <not in database>
                                            affinity ( RW): <not in database>
                                        other-config (MRW): import_task: OpaqueRef:d0c0d5fe-8037-4f4e-a042-81c582f1eada; mac_seed: e4c5e202-c4f5-2807-c12b-3258fa268e27; xo:backup:datetime: 20210718T09:30:33Z; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom
                                              dom-id ( RO): -1
                                     recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                                       xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data:
                          ha-always-run ( RW) [DEPRECATED]: false
                                 ha-restart-priority ( RW):
                                               blobs ( RO):
                                          start-time ( RO): 19700101T00:00:00Z
                                        install-time ( RO): 19700101T00:00:00Z
                                        VCPUs-number ( RO): 0
                                   VCPUs-utilisation (MRO):
                                          os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0
                                  PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42
                  PV-drivers-up-to-date ( RO) [DEPRECATED]: true
                                              memory (MRO):
                                               disks (MRO):
                                                VBDs (SRO): 7d8d958b-d20c-eb47-5229-bc80a988e1ad; e8176121-0637-c55e-61b5-e4b57fef14be
                                            networks (MRO): 0/ip: 169.254.10.199; 0/ipv4/0: 169.254.10.199; 0/ipv6/0: fe80:0000:0000:0000:84ec:876c:b5e6:0ac7
                                 PV-drivers-detected ( RO): true
                                               other (MRO): shutdown: poweroff; feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1
                                                live ( RO): true
                          guest-metrics-last-updated ( RO): 20210719T17:00:36Z
                                 can-use-hotplug-vbd ( RO): true
                                 can-use-hotplug-vif ( RO): true
                            cooperative ( RO) [DEPRECATED]: true
                                                tags (SRW): Disaster Recovery
                                           appliance ( RW): <not in database>
                                   snapshot-schedule ( RW): <not in database>
                                    is-vmss-snapshot ( RO): false
                                         start-delay ( RW): 0
                                      shutdown-delay ( RW): 0
                                               order ( RW): 0
                                             version ( RO): 0
                                       generation-id ( RO): 4856085469031164830:1232191562809696746
                           hardware-platform-version ( RO): 2
                                   has-vendor-device ( RW): true
                                     requires-reboot ( RO): false
                                     reference-label ( RO): windows-server-2016-64bit
                                        bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d
              
              
              fohdeeshaF 1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Oh okay, you are using disaster recovery and not continuous replication 🙂 Sorry I missed that, despite it's in the title 😛

                So there's no blocked operations in DR, at least it seems.

                1 Reply Last reply Reply Quote 0
                • julien-fJ Offline
                  julien-f Vates 🪐 Co-Founder XO Team @bazzacad
                  last edited by

                  @bazzacad The replicated VM has its start operation blocked which means that it cannot automatically power-on.

                  But, if you force started this replicated VM via XO, it removed the blockage and now the VM will automatically power-on.

                  1 Reply Last reply Reply Quote 0
                  • fohdeeshaF Offline
                    fohdeesha Vates 🪐 Pro Support Team @bazzacad
                    last edited by

                    I just confirmed with tests that Disaster Recovery jobs do indeed set the blocked-operations parameter for backed up VMs. As Julien said, did you ever force-start this VM? That would have removed start from the blocked-operations field, which is why we don't see it in your param-list, and why it's now being allowed to autostart.

                    B 1 Reply Last reply Reply Quote 0
                    • B Offline
                      bazzacad @fohdeesha
                      last edited by

                      @fohdeesha
                      Thanks for the help everyone. I'm 100% sure I did not force start this VM. In fact all of the DR VMs were running. I know my screen shot it only shows 1 running, but I killed the others ASAP before taking the screen shot.
                      What I think happened is the secondary host crashed, so I rebooted it.
                      And I see this parameter is set: actions-after-crash ( RW): Restart.
                      Is it possible that when the host recovers from a crash it sends the signal to the VMs to force restart? I sent you the parameter of the DR VM that did start up date 2021-07-18 (above), but if I list the parameter from the DR VM for the next day 2021-07-19 the block parameter is set:

                      [07:48 VULCAN ~]# xe vm-param-list uuid=e327b268-887d-395a-9665-2509c341df21
                      uuid ( RO)                                  : e327b268-887d-395a-9665-2509c341df21
                                                  name-label ( RW): rivieraVM2 - Daily Disaster Recovery - (20210719T165550Z)
                                            name-description ( RW): Windows Server 2016 License Manager & Print Server
                                                user-version ( RW): 2
                                               is-a-template ( RW): false
                                         is-default-template ( RW): false
                                               is-a-snapshot ( RO): false
                                                 snapshot-of ( RO): <not in database>
                                                   snapshots ( RO):
                                               snapshot-time ( RO): 19700101T00:00:00Z
                                               snapshot-info ( RO): disk-snapshot-type: crash_consistent; power-state-at-snapshot: Running
                                                      parent ( RO): <not in database>
                                                    children ( RO):
                                           is-control-domain ( RO): false
                                                 power-state ( RO): halted
                                               memory-actual ( RO): 0
                                               memory-target ( RO): 0
                                             memory-overhead ( RO): 73400320
                                           memory-static-max ( RW): 8589934592
                                          memory-dynamic-max ( RW): 8589934592
                                          memory-dynamic-min ( RW): 2147483648
                                           memory-static-min ( RW): 1073741824
                                            suspend-VDI-uuid ( RW): <not in database>
                                             suspend-SR-uuid ( RW): <not in database>
                                                VCPUs-params (MRW): weight: 256
                                                   VCPUs-max ( RW): 4
                                            VCPUs-at-startup ( RW): 4
                                      actions-after-shutdown ( RW): Destroy
                                        actions-after-reboot ( RW): Restart
                                         actions-after-crash ( RW): Restart
                                               console-uuids (SRO):
                                                         hvm ( RO): false
                                                    platform (MRW): timeoffset: -25200; videoram: 8; hpet: true; device-model: qemu-upstream-compat; apic: true; device_id: 0002; cores-per-socket: 2; pae: true; vga: std; nx: true; viridian_time_ref_count: true; viridian: true; acpi: 1; viridian_reference_tsc: true
                                          allowed-operations (SRO): changing_NVRAM; changing_dynamic_range; changing_shadow_memory; changing_static_range; make_into_template; migrate_send; destroy; export; start_on; start; clone; copy; snapshot
                                          current-operations (SRO):
                                          blocked-operations (MRW): start: Start operation for this vm is blocked, clone it if you want to use it.
                                         allowed-VBD-devices (SRO): 1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30; 31; 32; 33; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47; 48; 49; 50; 51; 52; 53; 54; 55; 56; 57; 58; 59; 60; 61; 62; 63; 64; 65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 91; 92; 93; 94; 95; 96; 97; 98; 99; 100; 101; 102; 103; 104; 105; 106; 107; 108; 109; 110; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 220; 221; 222; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254
                                         allowed-VIF-devices (SRO): 1; 2; 3; 4; 5; 6
                                              possible-hosts ( RO): 41b7c179-17c1-41f9-8f4c-31e92fd1271e
                                                 domain-type ( RW): hvm
                                         current-domain-type ( RO): unspecified
                                             HVM-boot-policy ( RW): BIOS order
                                             HVM-boot-params (MRW): order: ndc
                                       HVM-shadow-multiplier ( RW): 1.000
                                                   PV-kernel ( RW):
                                                  PV-ramdisk ( RW):
                                                     PV-args ( RW):
                                              PV-legacy-args ( RW):
                                               PV-bootloader ( RW):
                                          PV-bootloader-args ( RW):
                                         last-boot-CPU-flags ( RO): vendor: GenuineIntel; features: 1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000
                                            last-boot-record ( RO): ''
                                                 resident-on ( RO): <not in database>
                                                    affinity ( RW): <not in database>
                                                other-config (MRW): xo:backup:sr: bbdc9045-8262-19de-53f6-3f1f448a06c7; import_task: OpaqueRef:1106f8b8-3ade-47c6-981f-b8c7a2100f0c; mac_seed: 229f9501-ea5d-79f5-b2ba-dac4f972a558; xo:backup:schedule: 400780ea-631c-4489-9dc1-ed55cb50b13c; xo:backup:job: 61f64af8-caf3-4a01-8de9-67a17c8ca732; xo:backup:vm: 71d0babb-2450-417b-d60c-f9966cded4e2; xo:backup:datetime: 20210719T16:55:50Z; auto_poweron: true; base_template_name: Windows Server 2016 (64-bit); install-methods: cdrom
                                                      dom-id ( RO): -1
                                             recommendations ( RO): <restrictions><restriction field="memory-static-max" max="1649267441664"/><restriction field="vcpus-max" max="32"/><restriction field="has-vendor-device" value="true"/><restriction field="allow-network-sriov" value="1"/><restriction max="255" property="number-of-vbds"/><restriction max="7" property="number-of-vifs"/></restrictions>
                                               xenstore-data (MRW): vm-data/mmio-hole-size: 268435456; vm-data:
                                  ha-always-run ( RW) [DEPRECATED]: false
                                         ha-restart-priority ( RW):
                                                       blobs ( RO):
                                                  start-time ( RO): 19700101T00:00:00Z
                                                install-time ( RO): 19700101T00:00:00Z
                                                VCPUs-number ( RO): 0
                                           VCPUs-utilisation (MRO):
                                                  os-version (MRO): name: Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition2; distro: windows; major: 10; minor: 0; spmajor: 0; spminor: 0
                                          PV-drivers-version (MRO): major: 9; minor: 0; micro: 1; build: 42
                          PV-drivers-up-to-date ( RO) [DEPRECATED]: true
                                                      memory (MRO):
                                                       disks (MRO):
                                                        VBDs (SRO): 1b916155-4bc6-55da-df30-e115d913fff8; 315b8d45-14d6-79bd-4582-9b7a4820d120
                                                    networks (MRO): 0/ip: 192.168.254.30; 0/ipv4/0: 192.168.254.30; 0/ipv6/0: fe80:0000:0000:0000:7c2e:a532:d498:8b65
                                         PV-drivers-detected ( RO): true
                                                       other (MRO): feature-wuRebootRequired: 1; feature-static-ip-setting: 1; feature-ts: 1; feature-ts2: 1; feature-xs-batcmd: 1; feature-setcomputername: 1; error: WTSQueryUserToken : 1008 failed.; feature-balloon: 1; feature-suspend: 1; has-vendor-device: 1; platform-feature-multiprocessor-suspend: 1; data-ts: 1
                                                        live ( RO): true
                                  guest-metrics-last-updated ( RO): 20210718T15:56:35Z
                                         can-use-hotplug-vbd ( RO): true
                                         can-use-hotplug-vif ( RO): true
                                    cooperative ( RO) [DEPRECATED]: true
                                                        tags (SRW): Disaster Recovery
                                                   appliance ( RW): <not in database>
                                           snapshot-schedule ( RW): <not in database>
                                            is-vmss-snapshot ( RO): false
                                                 start-delay ( RW): 0
                                              shutdown-delay ( RW): 0
                                                       order ( RW): 0
                                                     version ( RO): 0
                                               generation-id ( RO): 7231909492726054475:6711832078023216933
                                   hardware-platform-version ( RO): 2
                                           has-vendor-device ( RW): true
                                             requires-reboot ( RO): false
                                             reference-label ( RO): windows-server-2016-64bit
                                                bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d
                      
                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by olivierlambert

                        No, a crash wouldn't trigger that. The only way to have this happen is:

                        • to force start the VM in XO
                        • or remove the start from blocked_operations elsewhere, eg via a xe

                        But the first one seems more plausible 🙂

                        action-after-crash doesn't have a related meaning (it's just that if your VM crashes, it will be restarted by Xen automatically). It's not host related.

                        B 1 Reply Last reply Reply Quote 0
                        • B Offline
                          bazzacad @olivierlambert
                          last edited by

                          @olivierlambert
                          Well I just rebooted my secondary host again via iDrac & these DR VMs started up again.
                          Note, it started 2 of the same one. And I'm 110% sure I didn't force start any of these, just rebooted the host.

                          248a113a-69f7-4804-a6dd-eff91e281184-image.png

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            And those two got any blocked operations in their record?

                            B 1 Reply Last reply Reply Quote 0
                            • B Offline
                              bazzacad @olivierlambert
                              last edited by

                              @olivierlambert
                              No, the 2 DR VMs that auto-started, don't have the blocked-operations set, the other DR VMs from other days do have it set.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                That's the issue. Why they don't have it is the problem/mystery.

                                Try to check if new replicated VMs got it. If not, I would check if you are correctly up to date.

                                1 Reply Last reply Reply Quote 0
                                • B bogdantomasciuc referenced this topic on
                                • First post
                                  Last post