This reminds me that I have some work to do on my policy, mostly for anti-affinity.

Hi @simonp, thanks again. Great service!! I have checked out your fix branch and re-run the backup. It succeeded as seen here: { "data": { "mode": "delta", "reportWhen": "always" }, "id": "1772725052650", "jobId": "73d72ea9-f25e-4c88-878c-98af27c1a827", "jobName": "Daily backup of Unix VMs with changing data", "message": "backup", "scheduleId": "9cbc3238-382d-4fae-83c0-4e92348898e5", "start": 1772725052650, "status": "success", "infos": [ { "data": { "vms": [ "c54a256e-0493-5c6d-a0cd-c3a4618f99a1" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c54a256e-0493-5c6d-a0cd-c3a4618f99a1", "name_label": "MailCW" }, "id": "1772725054418", "message": "backup VM", "start": 1772725054418, "status": "success", "tasks": [ { "id": "1772725054426", "message": "clean-vm", "start": 1772725054426, "status": "success", "tasks": [ { "id": "1772725055044", "message": "merge", "start": 1772725055044, "status": "success", "end": 1772725196294 } ], "end": 1772725196304, "result": { "merge": true } }, { "id": "1772725197030", "message": "snapshot", "start": 1772725197030, "status": "success", "end": 1772725198708, "result": "e46c757e-9180-2502-81c5-36a2fb2928fc" }, { "data": { "id": "fd6bbd08-52dd-43eb-84df-f93b54fc28e9", "isFull": false, "type": "remote" }, "id": "1772725198708:0", "message": "export", "start": 1772725198708, "status": "success", "tasks": [ { "id": "1772725200147", "message": "transfer", "start": 1772725200147, "status": "success", "end": 1772725224785, "result": { "size": 2768240640 } }, { "id": "1772725225673", "message": "clean-vm", "start": 1772725225673, "status": "success", "end": 1772725225747, "result": { "merge": true } } ], "end": 1772725225757 } ], "end": 1772725225758 } ], "end": 1772725225759 }

No worries, it happens! Glad you found the problem

I insist on the fact that this will only have impact on the FULL backups jobs , and not on the incremental ones

@DustyArmstrong said: @TeddyAstie yarp. My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request! 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 38 Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at d000 [size=256] Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable- Count=4 Masked- Vector table: BAR=5 offset=00042000 PBA: BAR=5 offset=00043000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000100, Page Request Allocation: 00000000 Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [400 v1] #25 Capabilities: [410 v1] #26 Capabilities: [440 v1] #27 Kernel driver in use: pciback thanks. So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly. That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools). I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

@florent I can try running this command next time memory usage is high and will report my findings!

@Greg_E Thanks, I've got another thread up and it's potentially being addressed!

@firefly because the underlying hardware that the VM has registered has likely changed, maybe substantially. A sysprep has the Windows system go through and validate what it's hardware is, it removes hardware specific drivers namely, but it does other stuff too.

And to really round this out, the MTBF for any of these is in the millions of hours (1.2-3M), that's a use time of 136.968 - 342.46 years respectively. Basically, if a drive dies, just replace it no matter what, but in the end the reliability of these drives is meant to outlast all of us. Unless you actually need some specific function provided in some form-factor or model, don't bother.

@DustinB doesn't it use the exact same mechanism? I have to find out.

@acebmxer I’m sorry to say that if those Dells are at your workplace, the wrong edition of iDRAC was purchased. You see with at the very least iDRAC 9 access to the full granular IPMI sensor data, was placed behind an edition paywall by Dell Technologies. Outlet temperature is just one of the feeds, missing from the Enterprise or lower edition of iDRAC 9. You’ll get the temperature readings from the Dell iDRAC web browser based interface, but not IPMI with iDRAC 9 Enterprise. To obtain the full IPMI sensor data you need the Datacenter edition.

@olivierlambert said: I think it was fixed since, are you sure you are using an uptodate XO? Commit 0be23 Anyway was directing attention to the GitHub issue, which was opened. The issue has another reference to it.

Good question! Ping @pdonias I assume

Latest XO can be new tested along XCP-ng (with OpenSSL 3): https://xcp-ng.org/forum/topic/9964/xcp-ng-8-3-updates-announcements-and-testing/363 Feedback welcome

@ItzJay in your journalctl -i xo-server log you should have something like nbdkit logs of ${diskPath} are in ${tmpDir} where tmpDir is something like /tmp/xo-serverXXXX/stderr can you post the file ? alternatively, you can open a support ticket

thanks Andryi We us round robin when using NBD , but to be fair, it does not change the performance a lot in most of the case. The concurrency settings ( multiple connection to the same file ) is helping when there is a high latency between XO and the host. SO , @fcgo if you have thousand of VMs , you should enable NBD it will consume less resource on the DOM0 and XO , and it will be spread on all the possible hosts.

@Forza Too funny. I came across this post and clicked on the URL you referenced......and that earlier question was from me! Well, nothing has changed. I'm doing mirror'ed backups and I'm still blind as a bat.