@ravenet well... thanks for the idea. I went ahead and tried spinning up an opensuse slowroll VM. I installed docker and the amdgpu firmware/kernel drivers, and tried running lemonade-server and ollama and same behavior. So that definitely seems to point to something weird going on on my host rather than the guest software stack...

Argh...Correction... 82 shows up as 82:01 and 82:02... lspci -vv -s 0000:82:01.0 82:01.0 PCI bridge: Intel Corporation Device e2f0 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=83, subordinate=83, sec-latency=0 Memory behind bridge: f7c00000-f7dfffff Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+ LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport And 82:02... lspci -vv -s 0000:82:02.0 82:02.0 PCI bridge: Intel Corporation Device e2f1 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=84, subordinate=84, sec-latency=0 Memory behind bridge: f7e00000-f7efffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport

Well you both just made my Friday! I can follow directions after all!

@tuxen Doing some research, it doesn't look like the Xeon's I have are affected. But I'm willing to try the next time I need to reboot. Will report back after that.

Hello, Thanks for all !

@laszlobortel Hehe yeah, those are pretty old kernels. I'd say there's a good chance kernel upgrades will go a long way to alleviating the CPU hangs. I can't say it's exclusively a problem with lvmohba storage, that's just what we use because of our previous VMware infrastructure was block storage over fiber channel. We knew a physical infra overhaul wasn't in the cards for us for this migration so we stayed with our existing storage system. This bug bit us half way through the migration until we figured out upgrading the kernel generally fixed it.

@psafont Thanks for the quick response and clarification. I appreciate you opening a work item for this. Looking forward to seeing this improvement in a future release.

For people running XCP-ng, wanting to use NUT client to power down gracefully during longer power outages, it could be interesting to note that Unifi now has two very affordable UPS models that feature a built in NUT server. So no need to run your own NUT server.

Hi, XCP-ng got an event system that will propagate things like this instantly, at least that's the way it works normally Do you have the same behaviour in Xen Orchestra? Have you reported the issue to CloudStack? If you have an XCP-ng support subscription, you can also open a ticket so we can take a look on XCP-ng status to catch any obvious issue.

@psafont Thank you for the quick response. I also found a similar issue: the other-config:auto-scan=true parameter is not being applied during xe sr-create either. As with the name-description parameter, the workaround is to add it separately afterwards using xe sr-param-add.

@julienXOvates Excellent, thanks for looking at this Julien! Rob

Thanks for that information. I will make this message short because @stormi is busy but I want to say thanks to Vates and XCP-ng for all their work done to support Windows on the Xen platform. This includes TPM2 and secure boot support and Microsoft-signed pv drivers. Well done!

The "PIF is not attached" usually means the network interface selected as the migration network isn't currently active on the target host. It can happen after upgrades if the host hasn't been fully rebooted. Worth checking whether a reboot of both hosts changes anything, and running xe pif-list to see whether that specific PIF shows currently-attached: true on the target. If the PIF looks attached in xe but migration still fails, might be worth a ping to Team-XAPI-Network.

perhaps "in the context of a proceeding RPU, do not start halted VMs" ? or "boot only halted VMs that have HA enabled" ? but I can imagine corner cases where this is not wanted. some chicken & egg problem.

I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine. After live migration the RAM of the linux guest is expanded to dynamic_max again. So this issue was in fact caused by Rust based xen-guest-agent. For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities. Best regards

@TeddyAstie Attached is the output you requested xen-cpuid -p.txt

@TeddyAstie That's more like it. I'm not finding the com2tcp though. I at least have something to search for. Once configured, is this persistent or do I need to create some sort of start up script that runs/loads a config on machine boot? Thanks!!

@abudef Some quotes from the documentation to clarify the situation: https://docs.xcp-ng.org/compute/#-nested-virtualization

Having help from OpenBSD side could be the way to go. Please forward your report to bugs@openbsd.org (with a link to this post too) For the record, I am able to reproduce it too (on XCP-ng 8.3 too, using OpenBSD -current).

@redakula said in Coral TPU PCI Passthrough: Frigate.nvr which is one of the popular uses for the Coral do not recommend it for new installs either. Frigate updated their recommendations because of the Google decision to sunset the device and because there are alternative options available to frigate for image inferencing. The coral is still supported though, and frigate is not the only use case or platform that can benefit from an accelerator. At the end of the day if you've already got the hardware, and it's efficient enough to run, then not using it is a waste or resources that could be allocated to other VMs.