Just tested, that was it. It was a 10 second fix to something I spent hours on. Well if anyone comes across this and is able to mount NFS storage pools successfully but cannot mount Backup Remotes, check that the Orchestra server itself (not the xcp-ng server) is on your NFS host ACL.
Thanks to everyone who took the time to read this and especially to those that responded.
No change after full xcp (8.2.1) patch and reboot. The VM in question started without issue as well.
For reference, I'm using XO commit d044d (currently 11 commits behind) on Master commit 6d34c.
I will leave as is since everything is functional to see if the future commit helps. I should be able to revisit at the end of the week to rebuild on the latest XO commit and test.
No "OLD_" VHDs in any SR found. Toolstack restart also did not resolve the issue. I will fully patch the server as the next step (probably should have done this earlier).
Awesome, great info! I don't recall seeing any atypical VDI names but I don't think I was looking specifically for that. I will re-check the SR and also restart the toolstack. The toolstack restart definitely holds some promise since this is a single server and, as a result, does not get restarted/patched often.
I should get a second server to be able to move critical workloads to (without downtime) and develop a more healthy patch/restart practice even if this is just a homelab. Thanks for that info!
@olivierlambert, if it sounds like this may be related to an issue being worked on and I can provide any helpful info, I'd be happy to gather anything that you think might be relevant.
I was doing some maintenance and noticed that the disk section of a running VM was empty. I have verified that somehow the primary vhd for this VM has the "is-a-snapshot" attribute set as true with no parent. It also seems that it is listed as a snapshot to itself:
xe vdi-param-list uuid=766d1995-19ba-420f-95e4-30e42dcbc698
uuid ( RO) : 766d1995-19ba-420f-95e4-30e42dcbc698
name-label ( RW): rss01 0
name-description ( RW):
is-a-snapshot ( RO): true
snapshot-of ( RO): 766d1995-19ba-420f-95e4-30e42dcbc698
snapshots ( RO): 6714273a-444b-4a21-ad58-b72abb85d6a7; 766d1995-19ba-420f-95e4-30e42dcbc698; 7b3e6fe8-a9f5-4cc1-8f13-d52f474bf7ab
snapshot-time ( RO): 20241228T06:08:27Z
allowed-operations (SRO): snapshot; clone
current-operations (SRO):
sr-uuid ( RO): e5eda81e-540b-029b-f180-20124f81163e
sr-name-label ( RO): HDD RAID1
vbd-uuids (SRO): 47e78473-706f-8e36-a017-c0983fdf2560
crashdump-uuids (SRO):
virtual-size ( RO): 214748364800
physical-utilisation ( RO): 426496
location ( RO): 766d1995-19ba-420f-95e4-30e42dcbc698
type ( RO): System
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): vhd-parent: 03b82421-c7a0-4c13-8d02-52aae2831674; read-caching-enabled-on-0250c976-1a99-4ee3-8b4b-27840941d478: true; host_OpaqueRef:24473335-4516-4675-aba9-ece2b4a46fef: RW
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
I found a couple of related posts. The first seems to have no resolution and it seems in the second, the solution was to export and import the VM:
https://xcp-ng.org/forum/topic/6981/vmguest-disk-is-missing-in-xen-orchestra/11
https://xcp-ng.org/forum/topic/6336/vm-missing-disk/26
I checked all SRs and none seem to have any coalesce locks so I'm not sure I have any current problems related to coalescence.
I'd like to resolve this in place if possible. Is there a way to set the "is-a-snapshot" flag to false? Is there something else I should check? Or is it best to export and import?
@julien-f
Awesome, thank you for the update. So if I'm hearing you correctly, this seems to be more of a minor annoyance and not necessarily anything harmful or detrimental. Thank you for verifying this is on the roadmap. I'll hang tight until it is patched. Thank you for all your work and efforts on this project!
@julien-f
I updated to commit 3410c today and the issue is still present.
@olivierlambert
I was originally using commit 1fac7. I reinstalled/upgraded today to 339d9 and the issue is still present. One interesting thing is these "started" tasks disappear when the page is refreshed.
@Darkbeldin
I started another topic because it seemed like there were some differences in my issue than was described here. However there are significant similarities too.
https://xcp-ng.org/forum/topic/7881/xo-user-authentication-excessive-and-unknown-tasks
I'm having this on xo-server 5.124.0, xo-client is 5.126.0 with a fully patched XCP-ng 8.2.1. You mention this being fixed recently. Any idea which specific version this was patched on and how it compares to the versions I listed?
This is very similar to the following post:
https://xcp-ng.org/forum/topic/7865/xo-tasks-xo-user-authentication-too-many
However there are a few differences that may warrant creating a new topic with reference to the old. Like the other topic, I am getting excessive "XO user authentication" tasks. However one difference is, when clicking on the "Open raw log" option, I just receive a "not found" page (not a 404 error but rather a page that simply displays the text "not found") and when I click cancel I receive a dialog from the cancel operation stating "Not Found". Additionally, most the tasks in question have a status of "started" and are yellow rather than green. Finally, I have performed a xcp-ng and xo server reboot with no change.
This is a new and freshly compiled XO instance from last night. xo-server is 5.124.0 and xo-client is 5.126.0. I'm on fully patched XCP-ng 8.2.1.
Did I do something wrong? Is this a known issue? Any thoughts on what might be causing it? Is there additional information I can include? I'm currently tailing some of the logs on the hypervisor including xensource.log but I'm not entirely sure what would be helpful.
Just tested, that was it. It was a 10 second fix to something I spent hours on. Well if anyone comes across this and is able to mount NFS storage pools successfully but cannot mount Backup Remotes, check that the Orchestra server itself (not the xcp-ng server) is on your NFS host ACL.
Thanks to everyone who took the time to read this and especially to those that responded.