Hey @ronan-a ,
What should I do to lower the chance of something in the past installation of xostor to affect my new installation?
lsblk is still showing the linstor volumes, vgs is also showing linstor_group.
Will a wipefs -af be enough? Or is the "Destroying SR" button in XOA is enough?
those two UUIDs are in the ./linstor-kv-tool-modified --dump-volumes -u 192.168.0.108 -g xcp-sr-linstor_group_thin_device device output
./linstor-kv-tool-modified --dump-volumes -u 192.168.0.108 -g xcp-sr-linstor_group_thin_device
{
"xcp/sr/journal/clone/0fb10e9f-b9ef-4b59-8b31-9330f0785514": "86b1b2af-8f1d-4155-9961-d06bbacbb7aa_0e121812-fcae-4d70-960f-ac440b3927e3",
"xcp/sr/journal/clone/14131ee4-2956-47b7-8728-c9790764f71a": "dfb43813-91eb-46b8-9d56-22c8dbb485fc_917177d5-d03b-495c-b2db-fd62d3d25b86",
"xcp/sr/journal/clone/45537c14-0125-4f6c-a1ad-476552888087": "36a23780-2025-4f3f-bade-03c410e63368_3e419764-9c8c-4539-9a42-be96f92e5c2a",
"xcp/sr/journal/clone/54ec7009-2424-4299-a9ad-fb015600b88c": "af89f0fc-7d5a-4236-b249-8d9408f5fb6d_f32f2e8f-a43f-43f5-824b-f673a5cbd988",
"xcp/sr/journal/clone/558220bc-a900-4408-a62e-a71a4bb4fd7b": "d9294359-c395-4bed-ac3a-bf4027c92bd9_0e18bf3d-78f0-4843-9e8f-ee11c6ebbf5a",
"xcp/sr/journal/clone/c41e0d47-5c1a-45c3-9404-01f3b5735c0d": "e191eb57-2478-4e3b-be9d-e8eaba8f9efe_41eae673-a280-439b-a4c6-f3afe2390fde",
"xcp/sr/journal/relink/50170fa2-2ca9-4218-8217-5c99ac31f10b": "1"
}
So I basically deleted all of the keys here, Maybe I should not have done that, but when I did, eva plugged in correctly to the SR and I was able to finally destroying the SR from XOA. So yeah happy ending. Will try the next beta version. Thank you @ronan-a for your work.
Figured out the issue, when I tried to update it from the cli instead. the /var/log partition was full due to /var/log/linstor-controller having something like 3.5G+ data (90% of the /var/log volume). maybe it is due to the past errors it accumulated. I deleted these logs and mike updated normally.
Now regarding plugging the PBD to eva (the one host that is not connecting to it). it says the following error.
Error code: SR_BACKEND_FAILURE_202
Error parameters: , General backend error [opterr=Base copy 36a23780-2025-4f3f-bade-03c410e63368 not present, but no original 45537c14-0125-4f6c-a1ad-476552888087 found],
this is what linstor resource-definition is showing
[03:59 eva ~]# linstor --controllers=192.168.0.108 resource-definition list -p
+---------------------------------------------------------------------------+
| ResourceName | Port | ResourceGroup | State |
|===========================================================================|
| xcp-persistent-database | 7000 | xcp-sr-linstor_group_thin_device | ok |
+---------------------------------------------------------------------------+
And here is the KV store for linstor from that script
[04:01 phoebe ~]# mountpoint /var/lib/linstor
/var/lib/linstor is a mountpoint
[04:01 phoebe ~]# ./linstor-kv-tool-modified --dump-volumes -u 192.168.0.108 -g xcp-sr-xcp-sr-linstor_group_thin_device
{
"xcp/sr/metadata": "{\"name_description\": \"\", \"name_label\": \"XOSTOR\"}"
}
[04:01 phoebe ~]# ./linstor-kv-tool-modified --dump-volumes -u 192.168.0.108 -g xcp-sr-linstor_group_thin_device
{
"xcp/sr/journal/clone/0fb10e9f-b9ef-4b59-8b31-9330f0785514": "86b1b2af-8f1d-4155-9961-d06bbacbb7aa_0e121812-fcae-4d70-960f-ac440b3927e3",
"xcp/sr/journal/clone/14131ee4-2956-47b7-8728-c9790764f71a": "dfb43813-91eb-46b8-9d56-22c8dbb485fc_917177d5-d03b-495c-b2db-fd62d3d25b86",
"xcp/sr/journal/clone/45537c14-0125-4f6c-a1ad-476552888087": "36a23780-2025-4f3f-bade-03c410e63368_3e419764-9c8c-4539-9a42-be96f92e5c2a",
"xcp/sr/journal/clone/54ec7009-2424-4299-a9ad-fb015600b88c": "af89f0fc-7d5a-4236-b249-8d9408f5fb6d_f32f2e8f-a43f-43f5-824b-f673a5cbd988",
"xcp/sr/journal/clone/558220bc-a900-4408-a62e-a71a4bb4fd7b": "d9294359-c395-4bed-ac3a-bf4027c92bd9_0e18bf3d-78f0-4843-9e8f-ee11c6ebbf5a",
"xcp/sr/journal/clone/c41e0d47-5c1a-45c3-9404-01f3b5735c0d": "e191eb57-2478-4e3b-be9d-e8eaba8f9efe_41eae673-a280-439b-a4c6-f3afe2390fde",
"xcp/sr/journal/relink/50170fa2-2ca9-4218-8217-5c99ac31f10b": "1"
}
I destroyed the PBD and then recreated it to make it just connect so I can destroy the SR, but the same error happened when I tried to connect with the new PBD that has the same config as the other PBD
@ronan-a I started updating xcp-ng so it can both restart and update on my four nodes (eva, phoebe, mike, ozly).
The nodes were updated with the rolling method, and all three node updated fine, except the forth (mike) (different that the ones that refuses to connect the PBD(eva)) since it is task was stuck at 0.000 progress for 3 hours, so i restarted the toolstack for it(mike) but it didnt do anything, so i restarted the master(eva) node stack. Then when I went to manually update it from XOA, it gives me this error.
-1(global name 'commmand' is not defined, , Traceback (most recent call last):
File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 101, in wrapper
return func(*args, **kwds)
File "/etc/xapi.d/plugins/updater.py", line 96, in decorator
return func(*args, **kwargs)
File "/etc/xapi.d/plugins/updater.py", line 157, in update
raise error
NameError: global name 'commmand' is not defined
)
The good news is, the linstor controller have moved to a different node(phoebe) from the old one(mike) and I was able to delete all volumes in the linstor --controllers=... resource-definition list except for the database, yet the PBD(eva) could not be connected. And the XOA still shows me a lot of disk, and when I scan it I get this error SR_HAS_NO_PBDS.
So now mike server cant update, and eva server cant connect its PBDs while all the other servers are connected. Note eva was the server that I started my linstor installation on.
Do you have any thoughts on what I can do to fix this without reinstalling xcp-ng on mike?
@ronan-a Thank you for your fast support.
I did these changes
diff -u linstor-kv-tool linstor-kv-tool-modified
--- linstor-kv-tool 2022-11-17 18:57:00.941259380 +0800
+++ linstor-kv-tool-modified 2022-11-17 19:04:15.957504667 +0800
@@ -33,7 +33,7 @@
kv = linstor.KV(
group_name,
uri=controller_uri,
- namespace='/xcp/volume/{}'.format(vdi_name)
+ namespace='/volume/{}'.format(vdi_name)
)
for key, value in list(kv.items()):
@@ -46,11 +46,11 @@
uri=controller_uri,
namespace='/'
)
-
for key, value in list(kv.items()):
- if key.startswith('xcp/volume/'):
+ if key.startswith('volume/'):
size = key.rindex('/')
kv.namespace = key[:size]
+ print("key is {}".format(repr(key[size + 1:])))
del kv[key[size + 1:]]
and I got the following error.
./linstor-kv-tool-modified --remove-all-volumes -u 192.168.0.106 -g xcp-sr-xcp-sr-linstor_group_thin_device
key is u'metadata'
Traceback (most recent call last):
File "./linstor-kv-tool-modified", line 78, in <module>
main()
File "./linstor-kv-tool-modified", line 74, in main
remove_all_volumes(args.uri, args.group_name)
File "./linstor-kv-tool-modified", line 54, in remove_all_volumes
del kv[key[size + 1:]]
File "/usr/lib/python2.7/site-packages/linstor/kv.py", line 151, in __delitem__
self._del_linstor_kv(k)
File "/usr/lib/python2.7/site-packages/linstor/kv.py", line 89, in _del_linstor_kv
raise linstor.LinstorError('Could not delete kv({}): {}'.format(k, rs[0]))
linstor.errors.LinstorError: Error: Could not delete kv(/volume/aec2104e-e501-4d7d-b0fb-95a80e843e0a/metadata): ERRO:Exception thrown.
and I can confirm the volume exist when I dump all of them
"volume/aec2104e-e501-4d7d-b0fb-95a80e843e0a/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": \"\", \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/aec2104e-e501-4d7d-b0fb-95a80e843e0a/not-exists": "0",
"volume/aec2104e-e501-4d7d-b0fb-95a80e843e0a/volume-name": "xcp-volume-b1748285-7cda-429f-b230-50dfba161e9c",
May I ask what do you recommend me to do? And thank you for your continues support.
@ronan-a Thank you for the detailed steps.
I get the following output when i dump my volumes using the xcp-sr-linstor_group_thin_device group
{
"xcp/sr/journal/clone/0fb10e9f-b9ef-4b59-8b31-9330f0785514": "86b1b2af-8f1d-4155-9961-d06bbacbb7aa_0e121812-fcae-4d70-960f-ac440b3927e3",
"xcp/sr/journal/clone/14131ee4-2956-47b7-8728-c9790764f71a": "dfb43813-91eb-46b8-9d56-22c8dbb485fc_917177d5-d03b-495c-b2db-fd62d3d25b86",
"xcp/sr/journal/clone/45537c14-0125-4f6c-a1ad-476552888087": "36a23780-2025-4f3f-bade-03c410e63368_3e419764-9c8c-4539-9a42-be96f92e5c2a",
"xcp/sr/journal/clone/54ec7009-2424-4299-a9ad-fb015600b88c": "af89f0fc-7d5a-4236-b249-8d9408f5fb6d_f32f2e8f-a43f-43f5-824b-f673a5cbd988",
"xcp/sr/journal/clone/558220bc-a900-4408-a62e-a71a4bb4fd7b": "d9294359-c395-4bed-ac3a-bf4027c92bd9_0e18bf3d-78f0-4843-9e8f-ee11c6ebbf5a",
"xcp/sr/journal/clone/c41e0d47-5c1a-45c3-9404-01f3b5735c0d": "e191eb57-2478-4e3b-be9d-e8eaba8f9efe_41eae673-a280-439b-a4c6-f3afe2390fde",
"xcp/sr/journal/relink/50170fa2-2ca9-4218-8217-5c99ac31f10b": "1"
}
but the --remove-all-volumes does not delete them because they dont start with xcp/volume/.
Also when i placed xcp-sr-xcp-sr-linstor_group_thin_device a lot of volumes appeared similar to the following
{
"volume/00897d74-53c9-41b4-8f5f-73132e4a9af7/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": null, \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/00897d74-53c9-41b4-8f5f-73132e4a9af7/not-exists": "0",
"volume/00897d74-53c9-41b4-8f5f-73132e4a9af7/volume-name": "xcp-volume-2892500d-d80a-4978-aa87-ab2b39ace9e9",
"volume/00b0dbb5-2dfa-4fd5-baf4-81065afa2431/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": null, \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/00b0dbb5-2dfa-4fd5-baf4-81065afa2431/not-exists": "0",
...
...
...
"volume/fcbcd0dc-8d90-441e-8d03-e435ac417b96/not-exists": "0",
"volume/fcbcd0dc-8d90-441e-8d03-e435ac417b96/volume-name": "xcp-volume-f3748b88-1b25-4f18-8f63-4017b09f2ac6",
"volume/fce3b2e0-1025-4c94-9473-e71562ca11bd/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": null, \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/fce3b2e0-1025-4c94-9473-e71562ca11bd/not-exists": "0",
"volume/fce3b2e0-1025-4c94-9473-e71562ca11bd/volume-name": "xcp-volume-08f1fb0b-d6a3-47eb-893b-6c8b08417726",
"volume/fe6bc8fd-4211-4b4a-8ee5-ba55a7641053/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": null, \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/fe6bc8fd-4211-4b4a-8ee5-ba55a7641053/not-exists": "0",
"volume/fe6bc8fd-4211-4b4a-8ee5-ba55a7641053/volume-name": "xcp-volume-7a46e0f4-0f61-4a37-b235-1d2bd9eaf033",
"volume/fe8dc6e6-a2c6-449a-8858-255a37cc8f98/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": \"\", \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/fe8dc6e6-a2c6-449a-8858-255a37cc8f98/not-exists": "0",
"volume/fe8dc6e6-a2c6-449a-8858-255a37cc8f98/volume-name": "xcp-volume-0290c420-9f14-43ae-9af5-fe333b60c7dc",
"volume/feadfc8d-5aeb-429c-8335-4530aa24cc86/metadata": "{\"read_only\": true, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": null, \"name_label\": \"base copy\", \"name_description\": \"\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}",
"volume/feadfc8d-5aeb-429c-8335-4530aa24cc86/not-exists": "0",
"volume/feadfc8d-5aeb-429c-8335-4530aa24cc86/volume-name": "xcp-volume-6a37cf38-d6e4-4af3-90d7-84bec3938b20",
"xcp/sr/metadata": "{\"name_description\": \"\", \"name_label\": \"XOSTOR\"}"
}
and the --remove-all-volumes also does not work on them.
I did the following with and without the xcp-sr prefix and it produced an empty json when specifiying the namesapce as /xcp/volume to match the startwith in the delete thingy.
./linstor-kv-tool --dump-volumes -u 192.168.0.106 -g xcp-sr-linstor_group_thin_device -n /xcp/volume
What do you think I should do?
@ronan-a I did not forget the SR so yeah it is the PBDs.
I tried to plug them back with
xe pbd-plug uuid=...
taking the uuid from the
xe pbd-list sr-uuid=xostor-uuid
I was able to plug three hosts, however the last forth host says the following.
Error code: SR_BACKEND_FAILURE_1200
Error parameters: , Cannot update volume uuid 36a23780-2025-4f3f-bade-03c410e63368 to 45537c14-0125-4f6c-a1ad-476552888087: this last one is not empty,
What do you think I should do to make the forth host pbd connect to delete the SR correctly?
Hey @ronan-a
I'm trying to remove the SR via XOA with "Remove this SR" button but im getting this error.
SR_NOT_EMPTY()
And there is no VM connected to it. So i tried to delete the disks manually in it but I get this error
SR_HAS_NO_PBDS(OpaqueRef:6d7520e0-60fa-4b93-9dfe-aa7ceb3b17d2)
Could you help me how to remove it? I dont care about its content so any way is okay for me since I want to reinstall it after i install the updates.
Hey @ronan-a , Now all the VDIs on the device are broken, I tried to migrate them but i get errors such as.
SR_BACKEND_FAILURE_1200(, Cannot update volume uuid 36a23780-2025-4f3f-bade-03c410e63368 to 45537c14-0125-4f6c-a1ad-476552888087: this last one is not empty, )
SR_BACKEND_FAILURE_78(, VDI Creation failed [opterr=error Error: Could not set kv(/volume/9cdc83cc-0fd8-490e-a3af-2ca40c95f398/not-exists:2): ERRO:Exception thrown.], )
SR_BACKEND_FAILURE_46(, The VDI is not available [opterr=Plugin linstor-manager failed], )
I dont care about the broken VDIs content so no worries.
It was fun experimenting with it, but I need more storage and will move the SSDs to my NAS and run my VMs on NFS there instead.
Before I do so I thought you might be interested in debugging the issues and getting my logs if that will help the project. Just let me know what files I need to send and will be happy to do so.
@ronan-a My appoiliges for replying late. The issue happened again and remembered this thread.
I tried cat /sys/kernel/debug/drbd/resources/xcp-volume-{UUID}/volumes/0/openers and it is empty across all hosts for both the old broken VDI and the new one.
The hosts are:
I also have scheduled backups snapshots so not sure if this will affect the vdi removal.
Here is the log SMlog.zip.txt The file is not a .txt it is just a .zip (the forum doesnt allow .zip).
The file is filled with bad volume and idk what to do to fix it.