Posts

• RE: Intermittent Xen blkfront I/O stalls: all guest tags busy while tapdisk reports zero outstanding requests

  Hello @mike.potapov , on n_reqs=32 and n_reqs_free=32, I think you're watching td_blktap_t structure. This structure is only used if the pvdriver are not install in the VM, so it's normal this ring is not used. You should watch struct td_xenblkif that is the ring use normally and that must contains 256 requests on max_order=3.

  The tapdisk will wakeup periodically, you could put a breakpoint on tapdisk_xenio_ctx_ring_event to confirm that the ring is check periodically. You can investigate td_xenblkif here.

  Ultimatly, you can try to send a notification from the tapdisk under gdb to the linux kernel with call (int)xenevtchn_notify(blkif->ctx->xce_handle, blkif->port); to see if it unlock something on the guest side. It it's the case it will confirm a notification was lost. If not you will get a message about spurious interrupt in the dmesg of the guest kernel.

  posted in Compute

  A

  anthoineb
  
• RE: Intermittent Xen blkfront I/O stalls: all guest tags busy while tapdisk reports zero outstanding requests

  Hello @kazual82 Sorry for the delay, I was working on a gdb script for you before I was preempt. Yes, I'm curious about GDB backtraces and ring dump. I'm puzzled because even if we miss a notification from the frontend, tapdisk periodically checks blkif ring, so he must see the request with a delay at least. Can you also share your daemon.log and SMlog?

  posted in Compute

  A

  anthoineb
  
• RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?

  @Jonathon Indeed I missed that part in your message, my bad.
  If the hosts still crash even after the 9.2.18 module has been loaded (update + reboot), please keep us informed.

  Thanks!

  posted in XOSTOR

  M

  Mathieu L
  
• RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?

  @Jonathon Have you rebooted after the update? I should have specified it but the kmod-drbd update is only effective after a reboot.

  If the 9.2.16 drbd module was still loaded (i.e. host not rebooted) 16GB would still have been filled quickly with the memory leak unfortunately. 16GB and 9.2.18 should be safe now though.

  If unsure you can check the output of modinfo drbd.

  posted in XOSTOR

  M

  Mathieu L
  
• RE: xcp-ng server crashed/rebooted due to issues with drbd/linstor?

  Hello,

  Indeed this is a known issue with XOSTOR, when causing a sync of too many resources at once it's possible for a DRBD resource to leak memory during the resync. A patch on the 9.2.18 fixes this by throttling the number of resources that can be synced at once to mitigate the issue.

  The 9.2.18 update should be in the main xcp-ng-linstor repo very soon.

  What you can do too is give more RAM to the dom0 of your hosts. How many RAM have been allocated to dom0 at the moment? We advise 16GB or 32GB depending on the number of volumes.

  posted in XOSTOR

  M

  Mathieu L
  
• RE: Intermittent Xen blkfront I/O stalls: all guest tags busy while tapdisk reports zero outstanding requests

  Hello @kazual82,
  Lot of question here, I'll try to answer my best:

  • Is there a known XCP-ng 8.3, blktap, or Xen issue matching this frontend/backend state mismatch?
    No, I never saw this kind of issue but I appreciate all your investigation.

  • Can tapdisk or the Xen backend lose a completion while both sides remain in state Connected?
    It'd surprise me because my opinion is tapdisk is sending to much completion notifications. My guess is a race condition somewhere that loose the notification.

  • Is xen_blkfront.max_ring_page_order=3 considered safe and recommended with this backend?
    Yes, it's safe but there is no recommendation on that because any max_ring_page_order should work.

  • Could increasing the ring size reduce the probability of this failure, or would it only mask it temporarily?
    I guess it reduce the probability.

  • Is the fact that persistent grants and indirect descriptors are not negotiated relevant?
    No, it's expected, tapdisk doesn't support those features.

  • Are there additional tapdisk, xenstore, event-channel, or ring diagnostics that we should enable before the next incident?
    I don't see what we can do. If it appends again I'm curious to investigate. Install blktap-debuginfo and devtoolset-11-gdb to attach the debugger on tapdisk and see the state of the blkif ring, that could be an info if the requests are still in the ring. If you can open a support tunnel, I will investigate.

  • Are there candidate packages or patches that would be useful to test on one canary host?
    The next update, yet to be release, will add a bunch of fixes for QCOW2 support in tapdisk and sm. I recommend this update.

  • Could the use of a multi-terabyte QCOW2 VDI on an LVM SR be relevant to this failure mode?
    I fear this is a QCOW2 issue only linked to the recent release of the feature but no real clue here.

  The fact that tap-ctl stats answer is the proof that there is no deadlock on the tapdisk side, that's a small reason of satisfaction.

  posted in Compute

  A

  anthoineb