@Danp

Well that's a kick in the pants. When I ran the script last week, it said it was updating node, but I didn't pay attention. Apparently, the update failed silently. Today it was noisy and ended up having to update node manually. We'll see how it goes. Looks like Roni is already aware of issues with his script where node is concerned.

@olivierlambert

I thought it was until I started poking at it this morning (was going to clear the cert along with the cookies and cache). Doesn't look like Roni's script takes care of that though, so maybe someday...

Thanks guys!

Just updated XO for the first time in a year and have been getting 'kicked out' of XO on the regular. Cleared cache and cookies. Tried private/incognito. No change.

Developer console not giving me much of anything, but here's a screenshot.

.

Sometimes it'll keep me logged in for hours, other times not very long at all. The latest, I logged in last night and it stayed logged in. Checked my backups and puttered around a bit this morning, but when I went back to it, I was logged out.

Tried finding anything useful in the server logs (journalctl & /var/log/messages) but nothing is jumping out at me. Here's the last few log entries (all entries after the backup job, before having to log back in):

Nov  9 07:30:20 xo.mydomain.com xo-server[949]: 2023-11-09T12:30:20.263Z xo:main INFO + Console proxy (troy - ::ffff:192.168.12.34)
Nov  9 07:30:20 xo.mydomain.com xo-server[949]: 2023-11-09T12:30:20.497Z xo:main INFO - Console proxy (troy - ::ffff:192.168.12.34)
Nov  9 07:30:21 xo.mydomain.com xo-server[949]: 2023-11-09T12:30:21.616Z xo:main INFO + Console proxy (troy - ::ffff:192.168.12.34)
Nov  9 07:40:26 xo.mydomain.com xo-server[949]: 2023-11-09T12:40:26.186Z xo:main INFO - Console proxy (troy - ::ffff:192.168.12.34)
Nov  9 07:40:31 xo.mydomain.com xo-server[949]: 2023-11-09T12:40:31.625Z xo:main INFO + Console proxy (troy - ::ffff:192.168.12.34)
Nov  9 08:15:55 xo.mydomain.com xo-server[949]: 2023-11-09T13:15:55.723Z xo:main INFO - Console proxy (troy - ::ffff:192.168.12.34)
Nov  9 12:08:24 xo.mydomain.com xo-server[949]: 2023-11-09T17:08:24.465Z xo:xo-server-auth-ldap INFO successfully bound as CN=Troy Settle,OU=Users,OU=MCD,DC=mydomain,DC=com => troy authenticated

If this is a potential bug, I'm more than happy to spend some time chasing it, but either way, I kind of need an idea of what to start looking at.

OS: Alma Linux 8.8
Node: 16.20.2 (updated with the XO installer script)
Previous XO: 7e4f4c from Nov 8, 2022
Current XO: db99a from Nov 3, 2023

Not sure what other info to provide or what other questions to ask.

Thanks!

@rjt It's been so long, I really don't remember any specifics. Other than following the GNS3 server installation docs on Debian 11 finally worked.

I've still not managed to get it to work with any of my 11th generation Dell servers. Pretty sure this comes down to CPU features. Fortunately, we're nearly done replacing all our 11th gen servers, so not a huge deal for us any more.

Home, however, is another story. I know 14th gen servers will work just fine for this, but they're still a little on the expensive side for the homelab, and I have no 12th or 13th gen servers to try on.

Good luck!

@olivierlambert

Was just about to edit my post. Finally remembered that ESC will show me the shutdown messages.

Hanging after "Stopped Systems Management Data Engine"

Yes. ISO storage is mounted from a TrueNAS VM, and now it makes sense.

Homelab, single host. Running a Dell R510. All VMs manually stopped.

When shutting down the host, the progress bar moves a bit then hangs. I've left it for up to 30 minutes before forcing a reboot via the iDrac.

How can I disable the graphical start/shutdown screens so I can see what processes are hanging? I've tried changing ttys, but am not seeing anything of interest.

FWIW, I've not seen this on my production cluster, which is running on R710s and R640s.

Thanks,

-Troy

@olivierlambert

So I'll leave this here for posterity... while the pre-built VM images don't seem to want to work, building my own image using Debian 11 is working great so far.

So, no problem with XCP-ng/Xen, just with the image built by the GNS3 team.

Thanks for the responses Oliver, much appreciated.

@olivierlambert

Yeah, have nested mode enabled. It is an older CPU (L5630), but seems to work just fine with other hypervisors. I'm finding very little information other than the GNS and/or EVE-ng team wanting to blame Xen for poor nested virtualization support.

I'm going to try again after I re-apply the 8.2 update. I really don't like posting without being up to date, but in this case, nested virtualization seems to be a fairly mature feature that hasn't changed much for a while.

Then again, maybe I'll just try it on my production pool, which is running more modern processors.

Still plugging away at this. I've patched XCP-ng 8.1 (had a failed upgrade to 8.2 that I'm also working on). Anyways, here's the xl dmesg following a crashed VM.

I've also tested nested virtualization by installing XCP-ng in a VM and installing a VM within that. Worked just fine.

Not sure what's next other than trying again after upgrading to 8.2.

(XEN) [00000065b5b3f240] Xen version 4.13.0-8.12.2 (mockbuild@[unknown]) (gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)) debug=n  Tue Jan 26 12:25:34 CET 2021
(XEN) [00000065b5b404e7] Latest ChangeSet: 85e1424de2dd, pq d251ff964c89
(XEN) [00000065b5b41f03] build-id: d518750b24eae6cc451da1475abde6b4ae390d93
(XEN) [00000065b5b4290f] Bootloader: GRUB 2.02
(XEN) [00000065b5b43486] Command line: dom0_mem=4304M,max:4304M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
(XEN) [00000065b5b4438f] Xen image load base address: 0xcea00000
(XEN) [00000065b5b44c0c] Video information:
(XEN) [00000065b5b45852]  VGA is graphics mode 640x480, 16 bpp
(XEN) [00000065b5b4654c]  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN) [00000065b5b4718f]  EDID info not retrieved because no DDC retrieval method detected
(XEN) [00000065b5b47caa] Disc information:
(XEN) [00000065b5b48443]  Found 11 MBR signatures
(XEN) [00000065b5b48c6d]  Found 6 EDD information structures
(XEN) [00000065b5b78a30] Xen-e820 RAM map:
(XEN) [00000065b5b79621]  0000000000000000 - 000000000009e000 (usable)
(XEN) [00000065b5b7a367]  0000000000100000 - 00000000cf379000 (usable)
(XEN) [00000065b5b7afdb]  00000000cf379000 - 00000000cf38f000 (reserved)
(XEN) [00000065b5b7bc89]  00000000cf38f000 - 00000000cf3ce000 (ACPI data)
(XEN) [00000065b5b7c903]  00000000cf3ce000 - 00000000d0000000 (reserved)
(XEN) [00000065b5b7d5c9]  00000000e0000000 - 00000000f0000000 (reserved)
(XEN) [00000065b5b7e1e1]  00000000fe000000 - 0000000100000000 (reserved)
(XEN) [00000065b5b7efa7]  0000000100000000 - 0000001030000000 (usable)
(XEN) [00000065b7c7ffa7] Kdump: 256MB (262144kB) at 0xbec00000
(XEN) [00000065b8232bc6] ACPI: RSDP 000F1530, 0024 (r2 DELL  )
(XEN) [00000065b82343c0] ACPI: XSDT 000F1634, 009C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b82366b0] ACPI: FACP CF3B3F9C, 00F4 (r3 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b8238c67] ACPI: DSDT CF38F000, 3B44 (r1 DELL   PE_SC3          1 INTL 20050624)
(XEN) [00000065b823a518] ACPI: FACS CF3B6000, 0040
(XEN) [00000065b823b540] ACPI: APIC CF3B3478, 015E (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b823ca9e] ACPI: SPCR CF3B35D8, 0050 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b823de6a] ACPI: HPET CF3B362C, 0038 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b823f1d2] ACPI: DMAR CF3B3668, 01D0 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b8240683] ACPI: MCFG CF3B38C4, 003C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b82419cc] ACPI: WD__ CF3B3904, 0134 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b8242d6d] ACPI: SLIC CF3B3A3C, 0024 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b82441e1] ACPI: ERST CF392CE4, 0270 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b824565b] ACPI: HEST CF392F54, 03A8 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b8246b8f] ACPI: BERT CF392B44, 0030 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b8247ee4] ACPI: EINJ CF392B74, 0170 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b8249283] ACPI: SRAT CF3B3BC0, 0370 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b824a721] ACPI: TCPA CF3B3F34, 0064 (r2 DELL   PE_SC3          1 DELL        1)
(XEN) [00000065b824bbc9] ACPI: SSDT CF3B7000, 3FD4 (r1  INTEL PPM RCM  80000001 INTL 20061109)
(XEN) [00000065b829bba7] System RAM: 65523MB (67095644kB)
(XEN) [00000065c2555dcf] SRAT: PXM 1 -> APIC 20 -> Node 0
(XEN) [00000065c2556ade] SRAT: PXM 2 -> APIC 00 -> Node 1
(XEN) [00000065c255752a] SRAT: PXM 1 -> APIC 22 -> Node 0
(XEN) [00000065c2557f21] SRAT: PXM 2 -> APIC 02 -> Node 1
(XEN) [00000065c255891e] SRAT: PXM 1 -> APIC 32 -> Node 0
(XEN) [00000065c2559306] SRAT: PXM 2 -> APIC 12 -> Node 1
(XEN) [00000065c2559ccc] SRAT: PXM 1 -> APIC 34 -> Node 0
(XEN) [00000065c255a69e] SRAT: PXM 2 -> APIC 14 -> Node 1
(XEN) [00000065c255b086] SRAT: PXM 1 -> APIC 21 -> Node 0
(XEN) [00000065c255ba86] SRAT: PXM 2 -> APIC 01 -> Node 1
(XEN) [00000065c255c355] SRAT: PXM 1 -> APIC 23 -> Node 0
(XEN) [00000065c255cd21] SRAT: PXM 2 -> APIC 03 -> Node 1
(XEN) [00000065c255d6ed] SRAT: PXM 1 -> APIC 33 -> Node 0
(XEN) [00000065c255e192] SRAT: PXM 2 -> APIC 13 -> Node 1
(XEN) [00000065c255eb2a] SRAT: PXM 1 -> APIC 35 -> Node 0
(XEN) [00000065c255f4d5] SRAT: PXM 2 -> APIC 15 -> Node 1
(XEN) [00000065c256158f] SRAT: Node 1 PXM 2 0-d0000000
(XEN) [00000065c2562515] SRAT: Node 1 PXM 2 100000000-830000000
(XEN) [00000065c25632a7] SRAT: Node 0 PXM 1 830000000-1030000000
(XEN) [000000666cc1a509] Domain heap initialised DMA width 32 bits
(XEN) [000000669621180f] vesafb: framebuffer at 0x00000000d5800000, mapped to 0xffff82c000201000, using 2048k, total 8192k
(XEN) [0000006696212a4f] vesafb: mode is 640x480x16, linelength=1280, font 8x8
(XEN) [00000066962139ad] vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
(XEN) [0000006696226309] CPU Vendor: Intel, Family 6 (0x6), Model 44 (0x2c), Stepping 2 (raw 000206c2)
(XEN) [00000066a11d8e1b] found SMP MP-table at 000fe710
(XEN) [00000066a13101f0] DMI 2.6 present.
(XEN) [00000066a13ff86a] Using APIC driver bigsmp
(XEN) [00000066a150836a] XSM Framework v1.0.0 initialized
(XEN) [00000066a1633e18] Initialising XSM SILO mode
(XEN) [00000066a175c2ed] ACPI: PM-Timer IO Port: 0x808 (32 bits)
(XEN) [00000066a18a649b] ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0]
(XEN) [00000066a1998683] ACPI:             wakeup_vec[cf3b600c], vec_size[20]
(XEN) [00000066a1b6e009] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x20] enabled)
(XEN) [00000066a1ce6621] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x00] enabled)
(XEN) [00000066a1e5e2ea] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x22] enabled)
(XEN) [00000066a1fd7bd8] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled)
(XEN) [00000066a2153e58] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x32] enabled)
(XEN) [00000066a22cc2e1] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x12] enabled)
(XEN) [00000066a24445aa] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x34] enabled)
(XEN) [00000066a25bc298] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x14] enabled)
(XEN) [00000066a2734489] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x21] enabled)
(XEN) [00000066a28ac4d8] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x01] enabled)
(XEN) [00000066a2a22b12] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x23] enabled)
(XEN) [00000066a2b990a1] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x03] enabled)
(XEN) [00000066a2d0f983] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x33] enabled)
(XEN) [00000066a2e86080] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x13] enabled)
(XEN) [00000066a2ffcb52] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x35] enabled)
(XEN) [00000066a317328c] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x15] enabled)
(XEN) [00000066a3333892] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
(XEN) [00000066a34f1ee1] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
(XEN) [00000066a367e392] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) [00000066a377ead2] ACPI: IOAPIC (id[0x01] address[0xfec80000] gsi_base[32])
(XEN) [00000066a3868f92] IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
(XEN) [00000066a398ded8] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [00000066a3a783d2] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [00000066a3b9042a] Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) [00000066a3cf75cf] ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) [00000066a3e55883] PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) [00000066a3f698d8] PCI: MCFG area at e0000000 reserved in E820
(XEN) [00000066a40c63e1] PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) [00000066a42bcdf0] Xen ERST support is initialized.
(XEN) [00000066a442b9d2] HEST: Table parsing has been initialized
(XEN) [00000066a4577e6a] Using ACPI (MADT) for SMP configuration information
(XEN) [00000066a46f3ac0] SMP: Allowing 32 CPUs (16 hotplug CPUs)
(XEN) [00000066a4847367] IRQ limits: 56 GSI, 3032 MSI/MSI-X
(XEN) [00000066a49f0583] Disabling C-states C3 and C6 on Nehalem Processors due to errata
(XEN) [00000066a4b04ec0] CPU0: Intel machine check reporting enabled
(XEN) [00000066a4c69a52] Speculative mitigation facilities:
(XEN) [00000066a4d9f418]   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN) [00000066a4f1ad12]   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) [00000066a5096d00]   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- SSBD-, Other: IBPB L1D_FLUSH BRANCH_HARDEN
(XEN) [00000066a5223ed8]   L1TF: believed vulnerable, maxphysaddr L1D 44, CPUID 40, Safe address 10000000000
(XEN) [00000066a74eb386]   Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN) [00000066a97c8192]   Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU
(XEN) [00000066abacc692]   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN) [00000066aff509f0]   PV L1TF shadowing: Dom0 disabled, DomU enabled
(XEN) [00000066b22956d2] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [00000066babf7c55] Platform timer is 14.318MHz HPET
(XEN) [    2.799022] Detected 2133.431 MHz processor.
(XEN) [    2.819484] alt table ffff82d08044ded0 -> ffff82d08045bf6a
(XEN) [    2.844328] [VT-D]Disabling Interrupt remapping due to Intel 5500/5520/X58 Chipset errata #47, #53
(XEN) [    2.878250] Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) [    2.895549] Intel VT-d Snoop Control enabled.
(XEN) [    2.912720] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    2.929925] Intel VT-d Queued Invalidation enabled.
(XEN) [    2.947045] Intel VT-d Interrupt Remapping not enabled.
(XEN) [    2.964151] Intel VT-d Posted Interrupt not enabled.
(XEN) [    2.981185] Intel VT-d Shared EPT tables not enabled.
(XEN) [    2.999892] I/O virtualisation enabled
(XEN) [    3.016721]  - Dom0 mode: Relaxed
(XEN) [    3.033382] Interrupt remapping disabled
(XEN) [    3.050016] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    3.067164] ENABLING IO-APIC IRQs
(XEN) [    3.083517]  -> Using old ACK method
(XEN) [    3.100015] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    4.017084] Allocated console ring of 64 KiB.
(XEN) [    4.033455] mwait-idle: max C-state 1 reached
(XEN) [    4.049730] VMX: Supported advanced features:
(XEN) [    4.065894]  - APIC MMIO access virtualisation
(XEN) [    4.081971]  - APIC TPR shadow
(XEN) [    4.097892]  - Extended Page Tables (EPT)
(XEN) [    4.113777]  - Virtual-Processor Identifiers (VPID)
(XEN) [    4.129733]  - Virtual NMI
(XEN) [    4.145408]  - MSR direct-access bitmap
(XEN) [    4.161133]  - Unrestricted Guest
(XEN) [    4.176670] HVM: ASIDs enabled.
(XEN) [    4.192231] HVM: VMX enabled
(XEN) [    4.207504] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    4.222997] HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) [    4.238985] alt table ffff82d08044ded0 -> ffff82d08045bf6a
(XEN) [    4.274806] mwait-idle: max C-state 1 reached
(XEN) [    4.310469] mwait-idle: max C-state 1 reached
(XEN) [    4.346093] mwait-idle: max C-state 1 reached
(XEN) [    4.381664] mwait-idle: max C-state 1 reached
(XEN) [    4.417176] mwait-idle: max C-state 1 reached
(XEN) [    4.452650] mwait-idle: max C-state 1 reached
(XEN) [    4.488078] mwait-idle: max C-state 1 reached
(XEN) [    4.523471] mwait-idle: max C-state 1 reached
(XEN) [    4.558737] mwait-idle: max C-state 1 reached
(XEN) [    4.593971] mwait-idle: max C-state 1 reached
(XEN) [    4.629141] mwait-idle: max C-state 1 reached
(XEN) [    4.664259] mwait-idle: max C-state 1 reached
(XEN) [    4.699519] mwait-idle: max C-state 1 reached
(XEN) [    4.734557] mwait-idle: max C-state 1 reached
(XEN) [    4.769482] mwait-idle: max C-state 1 reached
(XEN) [    4.783965] Brought up 16 CPUs
(XEN) [    4.799318] Testing NMI watchdog on all CPUs: ok
(XEN) [    4.891169] mcheck_poll: Machine check polling timer started.
(XEN) [    4.906677] mtrr: your CPUs had inconsistent variable MTRR settings
(XEN) [    4.922215] mtrr: probably your BIOS does not setup all CPUs.
(XEN) [    4.937775] mtrr: corrected configuration.
(XEN) [    4.953147] MTRR default type: uncachable
(XEN) [    4.968319] MTRR fixed ranges enabled:
(XEN) [    4.983506]   00000-9ffff write-back
(XEN) [    4.998000]   a0000-bffff uncachable
(XEN) [    5.011824]   c0000-cbfff write-protect
(XEN) [    5.025472]   cc000-d3fff write-back
(XEN) [    5.039186]   d4000-ebfff uncachable
(XEN) [    5.052690]   ec000-fffff write-protect
(XEN) [    5.066184] MTRR variable ranges enabled:
(XEN) [    5.079652]   0 base 0000000000 mask ff80000000 write-back
(XEN) [    5.093246]   1 base 0080000000 mask ffc0000000 write-back
(XEN) [    5.106857]   2 base 00c0000000 mask fff0000000 write-back
(XEN) [    5.120482]   3 base 0100000000 mask ff00000000 write-back
(XEN) [    5.134121]   4 base 0200000000 mask fe00000000 write-back
(XEN) [    5.147768]   5 base 0400000000 mask fc00000000 write-back
(XEN) [    5.161430]   6 base 0800000000 mask f800000000 write-back
(XEN) [    5.175114]   7 base 1000000000 mask ffc0000000 write-back
(XEN) [    5.188938]   8 disabled
(XEN) [    5.202472]   9 disabled
(XEN) [    5.215984] Dom0 has maximum 1016 PIRQs
(XEN) [    5.229487] csched_alloc_domdata: setting dom 0 as the privileged domain
(XEN) [    5.255990] NX (Execute Disable) protection active
(XEN) [    5.269672] *** Building a PV Dom0 ***
(XEN) [    5.617703]  Xen  kernel: 64-bit, lsb, compat32
(XEN) [    5.631320]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x302c000
(XEN) [    5.658697] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    5.672353]  Dom0 alloc.:   0000000824000000->0000000828000000 (1080567 pages to be allocated)
(XEN) [    5.699919]  Init. ramdisk: 000000102ecf7000->000000102ffffc8f
(XEN) [    5.714253] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    5.728192]  Loaded kernel: ffffffff81000000->ffffffff8302c000
(XEN) [    5.742393]  Init. ramdisk: 0000000000000000->0000000000000000
(XEN) [    5.756711]  Phys-Mach map: 0000008000000000->0000008000868000
(XEN) [    5.771158]  Start info:    ffffffff8302c000->ffffffff8302c4b8
(XEN) [    5.785629]  Xenstore ring: 0000000000000000->0000000000000000
(XEN) [    5.800142]  Console ring:  0000000000000000->0000000000000000
(XEN) [    5.814903]  Page tables:   ffffffff8302d000->ffffffff8304a000
(XEN) [    5.830592]  Boot stack:    ffffffff8304a000->ffffffff8304b000
(XEN) [    5.846355]  TOTAL:         ffffffff80000000->ffffffff83400000
(XEN) [    5.861665]  ENTRY ADDRESS: ffffffff8242b180
(XEN) [    5.877761] Dom0 has maximum 16 VCPUs
(XEN) [    5.952376] Must not mask UR signaling on 0000:00:00.0
(XEN) [    5.967080] Found masked UR signaling on 0000:00:01.0
(XEN) [    5.981850] Found masked UR signaling on 0000:00:03.0
(XEN) [    5.996598] Found masked UR signaling on 0000:00:07.0
(XEN) [    6.011376] Found masked UR signaling on 0000:00:09.0
(XEN) [    6.026231] Found masked UR signaling on 0000:00:0a.0
(XEN) [    6.041135] Masked VT-d error signaling on 0000:00:14.0
(XEN) [   10.563228] Initial low memory virq threshold set at 0x4000 pages.
(XEN) [   10.578387] Scrubbing Free RAM in background
(XEN) [   10.593389] Std. Loglevel: Errors, warnings and info
(XEN) [   10.608715] Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) [   10.624570] ***************************************************
(XEN) [   10.639956] Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
(XEN) [   10.655519] enabled.  Please assess your configuration and choose an
(XEN) [   10.670980] explicit 'smt=<bool>' setting.  See XSA-273.
(XEN) [   10.686456] ***************************************************
(XEN) [   10.701891] Booted on MLPDS/MFBDS-vulnerable hardware with SMT/Hyperthreading
(XEN) [   10.732031] enabled.  Mitigations will not be fully effective.  Please
(XEN) [   10.747802] choose an explicit smt=<bool> setting.  See XSA-297.
(XEN) [   10.763665] ***************************************************
(XEN) [   10.779625] 3... 2... 1...
(XEN) [   13.795757] Xen is relinquishing VGA console.
(XEN) [   13.906936] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) [   13.907211] Freed 600kB init memory
(XEN) [   19.846606] Must not mask UR signaling on 0000:00:00.0
(XEN) [   19.847180] Found masked UR signaling on 0000:00:01.0
(XEN) [   19.847759] Found masked UR signaling on 0000:00:03.0
(XEN) [   19.848356] Found masked UR signaling on 0000:00:07.0
(XEN) [   19.848952] Found masked UR signaling on 0000:00:09.0
(XEN) [   19.849535] Found masked UR signaling on 0000:00:0a.0
(XEN) [   19.850042] Masked VT-d error signaling on 0000:00:14.0
(XEN) [46146.555106] domain_crash called from vmx.c#vmx_update_guest_cr+0x516/0x6a0
(XEN) [46146.555109] Domain 9 (vcpu#1) crashed on cpu#4:
(XEN) [46146.555114] ----[ Xen-4.13.0-8.12.2  x86_64  debug=n   Not tainted ]----
(XEN) [46146.555116] CPU:    4
(XEN) [46146.555119] RIP:    0010:[<ffffffffc080d8f5>]
(XEN) [46146.555121] RFLAGS: 0000000000000002   CONTEXT: hvm guest (d9v1)
(XEN) [46146.555126] rax: 0000000000000020   rbx: 000000000175f000   rcx: 0000000080202001
(XEN) [46146.555128] rdx: 0000000006000000   rsi: 0000000000093df0   rdi: 0000000000000000
(XEN) [46146.555131] rbp: 0000000000100000   rsp: ffffb1e3c35c7c18   r8:  0000000000000000
(XEN) [46146.555133] r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) [46146.555134] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [46146.555137] r15: 0000000000000000   cr0: 0000000080050033   cr4: 0000000000002060
(XEN) [46146.555139] cr3: 000000010fe30006   cr2: 0000000000000000
(XEN) [46146.555140] fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) [46146.555143] ds: 0018   es: 0018   fs: 0018   gs: 0018   ss: 0018   cs: 0010

Wondering if anyone has had any luck getting the GNS3 VM to function under XCP-ng. I've tried both the vmware and hyper-v images and neither one wants to work, though each works flawlessly under player and hyper-v.

Google search has not been kind. The best I can come up with is something like "Xen has problems with nested virtualization," but nothing really all that helpful to get it working.

In short, any time I try to fire up QEMU, the intermediate VM crashes with no log entries or even a message from the kernel on the console.

Thanks,

-Troy

I'm sure I did something wrong, but not sure what...

Decided to update the pool yesterday. The whole thing went very smooth and we didn't drop so much as a phone call or database transaction.

Simply moved all the VMs around to empty a host, then the 2nd and 3rd. Went very smooth.

Today, I'm sitting here staring at the daily delta backup job, and it's been running for 11 hours and have transferred some 400GB so far. Normally, the daily backup takes 20-30 minutes and transfers between 15-30GB.

What did I mess up?

@olivierlambert

Well crud, just read the thread from a few days ago about "backups failing if one host is down."

Reading your responses to that, I now understand what happened. Even after changing masters, XO was still connecting to the old master, which re-directed it to the new master.

When the old master went offline, XO continued to function, but when the backup job was launched, it tried to connect to the IP of the old master.

I like that XO will redirect itself when connecting to a host rather than the master, but in the event that XO changes the master, shouldn't it be able to update it's own configuration?

I inadvertently fixed it, but didn't take any notes.

Changed master using XO. XO still sees all 3 hosts.

Detach old master, power down, fixed hardware issue, re-install XCP.

When I tried to add the host back into the pool, I found that it was still listed on the servers page, but saw no way to re-connect it, so I deleted it.

Now the entire pool is missing?

*** Now I know that servers and hosts are two completely different things. Odd nomenclature, but I can deal with it.

Re-added the new master server, now the pool is back. Added the old master back and joined it to the pool, and all is good.

I'm sure I missed something when changing masters and removing the old one, but not sure what. I would think that changing masters in XO would change all the things that needed to be changed. Guess not.

Had a hardware issue, so I changed master to another server in the pool, evacuated all the VMs and detached the host and removed it from the pool.

Backup jobs in XO started failing with an error about not being able to connect to the detached host: Error: connect EHOSTUNREACH 172.23.6.101:443.

Of course it's unreachable. The master was changed and the original server was removed from the pool and taken off line.

So, what did I miss? I know it's not a lot to go on, but this is all I'm seeing. Not sure where else to look.

@darkbeldin Guess you didn't follow... if I have 14 days worth of snapshots from XO stored on the NAS and the NAS has 14 days worth of snapshots for the dataset itself, I can then roll the NAS back 14 days, then the previous 14 days worth of XO snapshots are there.

None of that matters though. You say you're replicating as well. I assume you're using rsync or some other mechanism?

I like the idea of daily and weekly backups though. You storing all that in the same place(s) on your NAS or do you create separate datasets for each?

Started out with a daily backup job using Deltas and 14 days of retention.

Added ZFS replication between my TrueNAS boxes, which requires snapshots. If I use 14 days worth of snapshots, this gives me up to 28 days of backups.

I'm not saying that this is overkill, but it really does seem rather convoluted when you consider how it all works under the hood.

What is the proper balance here?

The deeper I go, the more I learn. Thanks for the straight forward answers!

When I'm done with my migration from Hyper-V, we'll have just 4 hosts, so doing it all manually isn't a huge issue.

Is this something that XOSAN would resolve?

I seem to be missing something, and Google is telling me there aren't any great matches. After scrolling through a few pages of results, I agree.

When starting the rolling update or trying to put a host into maintenance mode, I get these errors.

I can migrate VMs manually, so not quite sure what I'm missing or how to begin to troubleshoot this.

Thanks.