Slow SR Creation

irtaza9

Yes multipath is enabled and we have two target ips. When I scan the target it calls the sr.probeIscsiIqns method and return the iqns in the dropdown as *(ip,ip) and on the select of *(ip,ip) it calls sr.probeIscsiLuns method. Both method take time to return the response. 90+ LUNs will be listed in the Lun dropdown after 4-5 minutes. Then On selection of LUN with respect to the id it calls sr.probeIscsiExists to validate the LUN, this method also take 1-3 minutes. After all this on create button click if then calls a method sr.createIscsi . This takes too much time too. I have a pool of 3 hosts and all servers have verified connectives to the SAN Storage. If you need any information regarding this, I will be happy to share.

poddingue

Thanks for the numbers.
So the whole chain drags: 4 to 5 minutes to list the ~90 LUNs, another 1 to 3 to validate one, and the create on top of that.

From the XO side I can't tell where that time goes, XO itself, the host's storage layer, or the array answering slowly, and the one measurement that would split it is the host-side probe I mentioned: time xe sr-probe type=lvmoiscsi device-config:target=<ip>, then the same call with device-config:targetIQN=<iqn> added.

If the CLI takes just as long, XO is mostly off the hook. A 90-LUN multipath target doesn't sound exotic to me, so probes taking minutes feel worth a proper look, though I honestly don't know what's normal at that scale.

@Team-Storage, the numbers above are concrete now, in case it's worth a look on your side.

irtaza9

@poddingue I am facing a delay in PBD plug. I have a pool of 3 hosts, member hosts iscsi PBD is immediate but the master take so much time like 10,20 minutes. I want to make it fast. Also just to have info regarding pool master host, does master host perform any extra task as compared to member hosts which may cause the delay?

Goulven

@irtaza9 Hello, could you share the content of your logs during the SR creation with us to investigate on our side ?
You can produce a tar of logs with https://docs.xcp-ng.org/troubleshooting/log-files/#produce-a-status-report .

irtaza9

@Goulven I have shared the logs with you over the drive.

tjkreidl

@irtaza9 Is this on a single host or a pool? Are all hosts properly time-synchronized? It it related to just one specific storage device or on any, e.g., is the behavior the same on a locally-created SR? Are there any hung tasks causing unnecessary I/O on that storage device?

irtaza9

@tjkreidl

we face this in both cases either single or pool based.
Currently we are have 3 host wide pool and we probe a new LUN over iscsi for each disk
Yes, we have maintained NTP synchronization.
we have Huawei Backend in our storage systems and we face this in v3, v6, v7 which are different.
No Locally created are just fine.
In XO there are not task.

tjkreidl

@irtaza9 This is an iSCSI-based storage device? If so, is it on a totally separate network than the VMs and primary management interface?

irtaza9

@tjkreidl

Our network topology is:

MGMT Network

iLO/iDRAC
SSH

Data Network
SAN Network

tjkreidl

@irtaza9
Why iDRAC and XCP-ng Backups Are Slow (I hope this is helpful, assuming I understand your topology and configuration correctly):
• iDRAC Virtual Media / HTML5 Limits: iDRAC virtual media or browser-based streaming sessions are heavily throttled by encryption overhead and are not designed for bulk high-speed data exports. Furthermore, if the iDRAC dedicated network port is negotiated at 100 Mbps instead of 1 Gbps (check for an orange link light), transfer speeds will crawl.
• Xen Orchestra (XOA) / XAPI Bottlenecks: XCP-ng exports VM data through the XAPI stack by generating compressed .xva streams or full backups, which can heavily tax CPU cores on the XOA/dom0 appliance and cap throughput around 50–80 MiB/s even on fast networks. So, check with top or xentop what your dom0 load is when you have an active backup to see if the CPU and/ or memory are overburdened.
• CPU Starvation on XOA: Backup worker processes (backupWorker.mjs) can saturate available vCPUs on the Xen Orchestra appliance if it is only allocated 2 vCPUs. Again, check with top or xentop whetgher your vCPUS are saturated.