@florent I can try running this command next time memory usage is high and will report my findings!

@florent No problem, just thought it would be fun. Thanks for your work anyway!

@nikade said in Every virtual machine I restart doesn't boot.: @DustinB yeah im guessing the VDI isn't attached to the VM for some reason, based on the screenshot. Im also wondering if he ever rebooted the VM's after installing them with PXE Right, it's a likely answer... but even then I would've expected his PXE server to just restart the installation process all over again... assuming that the disk is attached to the VM etc and that PXE boot isn't disabled automatically like it is with an ISO after first boot. haha

Thanks for the heads-up and thanks for the help !

Also.. Thank you for taking a look at it and solving it so quickly, @florent It is a quite nice interaction when reporting bugs that they are looked at and triaged. Not saying everything needs to be a priority 1. But I do appreciate being close to the process. And if I can help make things better by reporting bugs, in a hopefully constructive way. Then I'll continue to contribute my way.

@MichaelCropper Truenas on bare metal just as storage, XCP-NG is also bare metal on 3 hosts to make a pool. That's the minimum if you want to enable High Availability, it also works really well as a "normal" pool which is what I have. Rolling Pool functions are great, click the button and the system moves the VMS off of the host that need updates, reboots, moves VMs off of the next host, repeat. Only works with Shared storage.

@semarie This pool is still on 8.2.1, we are trying to add this host in order upgrade with little to no downtime.

@pilow As I am completly new to this scripting and such I did reach out to Ai for assistance and believe the license check should be bypassed now. Again not suggested to use in production. Use at your own risk. still has its own branch for more testing before merging to main branch.

@olivierlambert It's even older than that. https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.v5.md#5540-2020-12-29 https://xen-orchestra.com/blog/xen-orchestra-5-54/

@DustinB said in Recommended DELL Hardware ?: Doesn't using non-shared storage ultimately cause the VM to be started on the other host? No. Live migration can occur between hosts even if: the VDI are located on non-shared storage the hosts are in completely different pools For the cross pool migration to succeed, both pools need to support a common instruction set. Otherwise, use the Warm Migration feature that does reboot the VM so that it is running the correct instruction set on the destination host.

@Danp alright, thanks. I'll give that a try.

@florent Thanks again for your assistance, greatly appreciated.

@acebmxer I’m sorry to say that if those Dells are at your workplace, the wrong edition of iDRAC was purchased. You see with at the very least iDRAC 9 access to the full granular IPMI sensor data, was placed behind an edition paywall by Dell Technologies. Outlet temperature is just one of the feeds, missing from the Enterprise or lower edition of iDRAC 9. You’ll get the temperature readings from the Dell iDRAC web browser based interface, but not IPMI with iDRAC 9 Enterprise. To obtain the full IPMI sensor data you need the Datacenter edition.

@benapetr said in New project - XenAdminQt - a cross-platform GNU/Linux, macOS, Windows native thick client: @Pilow I know them a little bit, I will have a look, but I am now working on another new cool thing! It's called xen_exporter: https://github.com/benapetr/xen_exporter It's a prometheus exporter that hooks directly to xen kernel via xenctrl library from dom0 and extract all low-level metrics from the host, allowing very detailed graphs with very low granularity with stuff I always missed in both XenOrchestra and XenAdmin: ... We have a similar project : https://github.com/xcp-ng/xcp-metrics, but unfortunately it's not used as of today (though it could get revived as Rust for Xen matures, i.e easier to build). There is also Xen Orchestra OpenMetrics support but it's not on XCP-ng itself.

@whyberg Nope... that feature is part of the CPU. But is that the actual problem? Maybe... As that machine is too old to do any real work, it's a fine toy. There are better faster cheap test machines available.

Hi @simonp, thanks again. Great service!! I have checked out your fix branch and re-run the backup. It succeeded as seen here: { "data": { "mode": "delta", "reportWhen": "always" }, "id": "1772725052650", "jobId": "73d72ea9-f25e-4c88-878c-98af27c1a827", "jobName": "Daily backup of Unix VMs with changing data", "message": "backup", "scheduleId": "9cbc3238-382d-4fae-83c0-4e92348898e5", "start": 1772725052650, "status": "success", "infos": [ { "data": { "vms": [ "c54a256e-0493-5c6d-a0cd-c3a4618f99a1" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c54a256e-0493-5c6d-a0cd-c3a4618f99a1", "name_label": "MailCW" }, "id": "1772725054418", "message": "backup VM", "start": 1772725054418, "status": "success", "tasks": [ { "id": "1772725054426", "message": "clean-vm", "start": 1772725054426, "status": "success", "tasks": [ { "id": "1772725055044", "message": "merge", "start": 1772725055044, "status": "success", "end": 1772725196294 } ], "end": 1772725196304, "result": { "merge": true } }, { "id": "1772725197030", "message": "snapshot", "start": 1772725197030, "status": "success", "end": 1772725198708, "result": "e46c757e-9180-2502-81c5-36a2fb2928fc" }, { "data": { "id": "fd6bbd08-52dd-43eb-84df-f93b54fc28e9", "isFull": false, "type": "remote" }, "id": "1772725198708:0", "message": "export", "start": 1772725198708, "status": "success", "tasks": [ { "id": "1772725200147", "message": "transfer", "start": 1772725200147, "status": "success", "end": 1772725224785, "result": { "size": 2768240640 } }, { "id": "1772725225673", "message": "clean-vm", "start": 1772725225673, "status": "success", "end": 1772725225747, "result": { "merge": true } } ], "end": 1772725225757 } ], "end": 1772725225758 } ], "end": 1772725225759 }

@DustyArmstrong said: @TeddyAstie yarp. My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request! 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 38 Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at d000 [size=256] Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable- Count=4 Masked- Vector table: BAR=5 offset=00042000 PBA: BAR=5 offset=00043000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000100, Page Request Allocation: 00000000 Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [400 v1] #25 Capabilities: [410 v1] #26 Capabilities: [440 v1] #27 Kernel driver in use: pciback thanks. So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly. That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools). I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

@Cyrille