SR_BACKEND_FAILURE_109

olivierlambert

Hi,

Yes, reviewing SMlog to understand why it's not coalescing. Or is it? Not enough space?

ntithomas

Aug 17 04:02:19 xenserv-R710-13 SM: [15610] sr_scan {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|77c12d43-e5dc-4275-8c6a-bb58cd961e9b|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:f6cd9a27-b540-4a03-bb86-6315bb044877', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:02:20 xenserv-R710-13 SM: [15730] sr_update {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|669eed1c-e621-4649-b7cd-ba6ac88d08b7|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:533ae798-6806-451b-b6ee-04eb60fd0db7', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:02:48 xenserv-R710-13 SM: [17059] sr_scan {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|283bc64c-9365-4a81-b1bb-7e962df8c86d|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:f1571087-9728-4815-981b-f843e7af01fc', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:02:49 xenserv-R710-13 SM: [17096] sr_update {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|c24de4bc-d286-4787-a9c1-65484fc4bf48|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:a4a2a0cd-6ff8-4a05-8b76-d74d7bcf4019', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:03:18 xenserv-R710-13 SM: [18533] sr_scan {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|c6bcdb0f-724f-49c2-a882-c3bfe4a8d384|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:3cee8e0c-f460-4e6f-866f-149d8103f722', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:03:19 xenserv-R710-13 SM: [18589] sr_update {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|a95985c3-bbec-459e-9573-b1071fd0540b|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:67a29d7e-94c4-4cc3-b446-87b6ae55fcbc', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:03:48 xenserv-R710-13 SM: [20029] sr_scan {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|662d165e-142e-48e2-be72-d83c9b421832|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:1d28ffee-cf9c-4da9-8392-30eff3d783b2', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:03:49 xenserv-R710-13 SM: [20055] sr_update {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|f2f6aafb-a08a-483d-b7cd-b3b8dc815803|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:66da129a-d70d-47be-a9ce-f6512b8413ad', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:04:19 xenserv-R710-13 SM: [21577] sr_scan {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|0f811b32-c7c4-4992-8986-b9bba823b53e|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:9f2ced8b-1932-4fc6-a7db-51e8a306cfee', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:04:19 xenserv-R710-13 SM: [21620] sr_update {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|fca6a164-fbed-451c-b65c-3f8bd98b233f|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:61a214d2-298c-42ed-9e1b-5120daccb210', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:04:49 xenserv-R710-13 SM: [23067] sr_scan {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|3be6c7a2-bdb5-45fe-b9fd-bdfb2a1e78c5|SR.scan', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:a56f34d9-23df-45ed-ae29-6a687443bbf7', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_scan', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}
Aug 17 04:04:49 xenserv-R710-13 SM: [23101] sr_update {'sr_uuid': '77c4b870-cfad-e713-a63e-3cdd2e3a8e29', 'subtask_of': 'DummyRef:|16b58459-c856-40d7-93e2-7d8ec833462f|SR.stat', 'args': [], 'host_ref': 'OpaqueRef:fdb8c160-29be-4e2d-8afe-0d5c3262ead5', 'session_ref': 'OpaqueRef:ebb247ca-9b75-4975-bff2-bd7733527215', 'device_config': {'username': 'louis', 'vers': '1.0', 'cifspassword_secret': '9d6a32d4-9519-ae39-4bed-30717ff3a91e', 'SRmaster': 'true', 'type': 'cifs', 'location': '//10.2.0.20/ISOs'}, 'command': 'sr_update', 'sr_ref': 'OpaqueRef:bc5a1530-c053-4dfa-99c3-fceee7ae3503'}

This is a copy of the varlog I believe. I am relatively new at this and was thrown into a pit of wolves basically.

ntithomas

I unfortunately am still learning how to read this log any help would be much appreciated!

olivierlambert

It's for your homelab, right? However, I have the feeling it's not. Don't tell me you have this issue in a production environment

ntithomas

Well unfortunately the previous guy put in a production environment. He left now I'm cleaning up his messes. So yes it is in a production environment I am currently trying to resolve the issues that were created by him.

olivierlambert

Ouch. And I suppose there's no pro support for this production environment?

ntithomas

Not that I am aware of. We have support individual VMs because they have paid software on them, but we don't have support on the VM manager itself.

olivierlambert

While we'll try to help, that would be a good time to tell people who write the checks to think about supporting this critical part of the infrastructure

Please provide the entire SMlog so we can take a look.

Danp

@ntinthomas I fixed your post above so that the log displays correctly. Please follow the below example in the future --

```
log goes here
```

ntithomas

The log is longer than i can post any suggestions to get it on here?

olivierlambert

Just download the entire file and host it somewhere

ntithomas

https://drive.google.com/file/d/1sbfZsUlONmtwKGNwhKLZPNdH5-v91EpQ/view?usp=sharing

I believe this is the entire log.

olivierlambert

In XO: SR view, click on the SR, advanced. Can you share a screenshot here?

Also, do you have VMs with a lot of snapshots?

ntithomas

This is the screen i generally see and work from. And yes we have about 13 VM's with screenshots being sent to a FreeNas box.

olivierlambert

XCP-ng center isn't an officially supported client, and worse than that, it doesn't expose the number of disks left to coalesce, unlike XO.

Also, you are on 8.1, which is EOL

ntithomas

So what would be the best solution to get this resolved?

olivierlambert

Do you have a lot of snapshots in your VMs?

Having an XOA deployed would help to diagnose too.

ntithomas

So we had them scheduled to take snapshots every night of every VM. I noticed that errors were accumulating this week. How would I deploy XOA to help diagnose the issue?

ntithomas

In the XCP-NG Center GUI we have 16 VM's with 41 snapshots currently in the snapshot section on the center.

olivierlambert

Using XCP-ng Center scheduled snapshot? That's really a poorly crafted feature vs XO backup capabilities (and it's not taking into account the chain, unlike XO).

I strongly suggest to get rid of XCP-ng Center at some point.

Regarding XOA deploy: https://help.vates.fr/kb/en-us/9-xen-orchestra-appliance-xoa/18-deploy-your-xoa

To fix your chain issues, please start to remove snapshots.