@PGibson
I run XO on a debian vm as well. However when I setup the vm, I installed it without any DE (desktop environment) so all the VM has is the shell.
I then access it using my client (desktop or laptop) browser. This works really well. I only have to ssh into the VM to update XO really (or do any other system maintenance). Otherwise pretty much everything is done via a browser.
I rebuilt XO to build 7579b with no change. I performed an export and import of the VM and the imported VM functioned normally and displayed the disk in the UI appropriately.
Just rebuilt XO up to commit a4986. The disk seems to be in the same state. Of course, if this commit includes the fix but the fix specifically prevents this issue, then I'll likely need to take additional steps to correct it to get back into a good state. Or, maybe that fix isn't yet in the commit (or maybe I have a situation that isn't addressed by the fix).
With regards to correcting the issue, I think there are a few options:
No change after full xcp (8.2.1) patch and reboot. The VM in question started without issue as well.
For reference, I'm using XO commit d044d (currently 11 commits behind) on Master commit 6d34c.
I will leave as is since everything is functional to see if the future commit helps. I should be able to revisit at the end of the week to rebuild on the latest XO commit and test.
No "OLD_" VHDs in any SR found. Toolstack restart also did not resolve the issue. I will fully patch the server as the next step (probably should have done this earlier).
Awesome, great info! I don't recall seeing any atypical VDI names but I don't think I was looking specifically for that. I will re-check the SR and also restart the toolstack. The toolstack restart definitely holds some promise since this is a single server and, as a result, does not get restarted/patched often.
I should get a second server to be able to move critical workloads to (without downtime) and develop a more healthy patch/restart practice even if this is just a homelab. Thanks for that info!
@olivierlambert, if it sounds like this may be related to an issue being worked on and I can provide any helpful info, I'd be happy to gather anything that you think might be relevant.
I was doing some maintenance and noticed that the disk section of a running VM was empty. I have verified that somehow the primary vhd for this VM has the "is-a-snapshot" attribute set as true with no parent. It also seems that it is listed as a snapshot to itself:
xe vdi-param-list uuid=766d1995-19ba-420f-95e4-30e42dcbc698
uuid ( RO) : 766d1995-19ba-420f-95e4-30e42dcbc698
name-label ( RW): rss01 0
name-description ( RW):
is-a-snapshot ( RO): true
snapshot-of ( RO): 766d1995-19ba-420f-95e4-30e42dcbc698
snapshots ( RO): 6714273a-444b-4a21-ad58-b72abb85d6a7; 766d1995-19ba-420f-95e4-30e42dcbc698; 7b3e6fe8-a9f5-4cc1-8f13-d52f474bf7ab
snapshot-time ( RO): 20241228T06:08:27Z
allowed-operations (SRO): snapshot; clone
current-operations (SRO):
sr-uuid ( RO): e5eda81e-540b-029b-f180-20124f81163e
sr-name-label ( RO): HDD RAID1
vbd-uuids (SRO): 47e78473-706f-8e36-a017-c0983fdf2560
crashdump-uuids (SRO):
virtual-size ( RO): 214748364800
physical-utilisation ( RO): 426496
location ( RO): 766d1995-19ba-420f-95e4-30e42dcbc698
type ( RO): System
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): vhd-parent: 03b82421-c7a0-4c13-8d02-52aae2831674; read-caching-enabled-on-0250c976-1a99-4ee3-8b4b-27840941d478: true; host_OpaqueRef:24473335-4516-4675-aba9-ece2b4a46fef: RW
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
I found a couple of related posts. The first seems to have no resolution and it seems in the second, the solution was to export and import the VM:
https://xcp-ng.org/forum/topic/6981/vmguest-disk-is-missing-in-xen-orchestra/11
https://xcp-ng.org/forum/topic/6336/vm-missing-disk/26
I checked all SRs and none seem to have any coalesce locks so I'm not sure I have any current problems related to coalescence.
I'd like to resolve this in place if possible. Is there a way to set the "is-a-snapshot" flag to false? Is there something else I should check? Or is it best to export and import?
@julien-f
Awesome, thank you for the update. So if I'm hearing you correctly, this seems to be more of a minor annoyance and not necessarily anything harmful or detrimental. Thank you for verifying this is on the roadmap. I'll hang tight until it is patched. Thank you for all your work and efforts on this project!
@julien-f
I updated to commit 3410c today and the issue is still present.
@olivierlambert
I was originally using commit 1fac7. I reinstalled/upgraded today to 339d9 and the issue is still present. One interesting thing is these "started" tasks disappear when the page is refreshed.
@Darkbeldin
I started another topic because it seemed like there were some differences in my issue than was described here. However there are significant similarities too.
https://xcp-ng.org/forum/topic/7881/xo-user-authentication-excessive-and-unknown-tasks
I'm having this on xo-server 5.124.0, xo-client is 5.126.0 with a fully patched XCP-ng 8.2.1. You mention this being fixed recently. Any idea which specific version this was patched on and how it compares to the versions I listed?
This is very similar to the following post:
https://xcp-ng.org/forum/topic/7865/xo-tasks-xo-user-authentication-too-many
However there are a few differences that may warrant creating a new topic with reference to the old. Like the other topic, I am getting excessive "XO user authentication" tasks. However one difference is, when clicking on the "Open raw log" option, I just receive a "not found" page (not a 404 error but rather a page that simply displays the text "not found") and when I click cancel I receive a dialog from the cancel operation stating "Not Found". Additionally, most the tasks in question have a status of "started" and are yellow rather than green. Finally, I have performed a xcp-ng and xo server reboot with no change.
This is a new and freshly compiled XO instance from last night. xo-server is 5.124.0 and xo-client is 5.126.0. I'm on fully patched XCP-ng 8.2.1.
Did I do something wrong? Is this a known issue? Any thoughts on what might be causing it? Is there additional information I can include? I'm currently tailing some of the logs on the hypervisor including xensource.log but I'm not entirely sure what would be helpful.
Just tested, that was it. It was a 10 second fix to something I spent hours on. Well if anyone comes across this and is able to mount NFS storage pools successfully but cannot mount Backup Remotes, check that the Orchestra server itself (not the xcp-ng server) is on your NFS host ACL.
Thanks to everyone who took the time to read this and especially to those that responded.
Before I head down that path, does that make sense? Is it the case that when you mount an nfs storage pool, Orchestra triggers xcp-ng to mount that which is why I could mount and create virtual disks. Whereas the Backup Remote is mounted by the Orchestra server rather than directly from xcp-ng? Or is that not the case...or do i just need to RTFM here?
I'm stupid. I forget that Orchestra is not running on the xcp-ng server (which I wish it did btw). I think I need to add the Orchestra server to the NAS allow list. Ugh, i can't believe how much time I spent on this for such an obvious oversight.
Is it the fact that I am running that with sudo in a terminal? Do I need to change something so Orchestra can do that without sudo?
I manually created the directory structure that the backup\remote store was trying to use and manually mounted the nfs to that. I verified that all the settings for the remote datastore matched. I can, in terminal, see the test data in the mountpoint. Running the backup with the NFS already mounted produces the same error.
Command failed with exit code 32: mount -o vers=3 -t nfs 172.23.1.20:/VM /run/xo-server/mounts/7f13127e-3167-4cc1-921a-6dd955984899 mount.nfs: access denied by server while mounting 172.23.1.20:/VM
Now I suspect if it needs to mount the store itself, it will fail if I already have it mounted to the location it wants to use. But what I don't get is, from a terminal, I can sucessfully run:
sudo mount -t nfs -o vers=3 172.23.1.20:/VM /run/xo-server/mounts/7f13127e-3167-4cc1-921a-6dd955984899
but it seems to me, this is what the backup is trying to run unsuccessfully. What am I missing here?
I'm sorry for the delay, life happens huh.
Anyway, I don't see it here but i seem to recall someone asking what if I try to mount it at the same location as the backup is. I get an error that the location doesn't exist. The path that the most recent backup I setup is trying to mount to is:
/run/xo-server/mounts/7f13127e-3167-4cc1-921a-6dd955984899
However that path does not exist. I do not have /run/xo-server. My run directory looks like this:
atd.pid mdadm rpc.statd.pid tapback.36
blktap-control message-switch samba tapback.4
boottime.stamp mount screen tapback.5
chrony mpathalert.pid sepermit tapback.6
chronyd.pid mpathcount.sock setrans tapback.8
console multipathd sm tapback.master
crond.pid netreport sm-notify.pid tapback.pid
cron.reboot net-snmp sr-mount tmpfiles.d
dbus nonpersistent sr-ref udev
faillock openvswitch sshd.pid usb-scan.sock
gssproxy.pid openvswitch.booted sudo user
gssproxy.sock perfmon.pid sysconfig utmp
initramfs plymouth syslogd.pid xapi_init_complete.cookie
iscsid.pid portreserve systemd xapi.pid
lldpad.pid portreserve.pid tapback.1 xapissl.pid
lock reboot-required.d tapback.10 xapi_startup.cookie
log rpcbind tapback.15 xen
lvm rpcbind.lock tapback.18 xenstored
mcelog-client rpcbind.sock tapback.2 xenstored.pid
mcelog.pid rpc.statd.lock tapback.35 xtables.lock
And yes, I built this from sources. The process went smooth with no anomalies. What's weird is I can mount this NFS as a normal datastore and create virtual disks on there without issue. I suspect this would address the permissions question. Does the Orchestra Backup feature use a different "user" than the rest of Orchestra? Did I maybe miss something in the build process?
I am having issues mounting an NFS Remote store for backups using XO. I'm on XCP-NG 8.1 with xo-server 5.66.0 and xo-web 5.69.0.
The error I receive is as follows:
Command failed with exit code 32: mount -o vers=3 -t nfs 172.23.1.20:/VM /run/xo-server/mounts/7f13127e-3167-4cc1-921a-6dd955984899 mount.nfs: access denied by server while mounting 172.23.1.20:/VM
I am able to manually mount from shell (including the ability to write) using
sudo mount -t nfs -o vers=3 172.23.1.20:/VM /mnt/nfs-test/
I also am able to successfully mount this as a standard nfs SR without issue in XO. This device is a QNAP so I've done some digging into the issues surrounding that such as:
https://github.com/xapi-project/sm/issues/511
But it seems as though this may not be the issue since showmount shows both columns (unless I missed something).
showmount -e 172.23.1.20
Export list for 172.23.1.20:
/Public *
/VM 172.23.1.17,172.23.1.16
I've also tried using vers=4 (which version 4 also works when mounting in XO as a standard nfs SR). What's weird with that is it seems to leave vers=3 in there and just appends ,vers=4 and I get a protocol error:
Command failed with exit code 32: mount -o vers=3,vers=4 -t nfs 172.23.1.20:/VM /run/xo-server/mounts/7f13127e-3167-4cc1-921a-6dd955984899 mount.nfs: Protocol not supported
Lastly, I've also created a folder at the root of the nfs share and tried to mount that but receive the same error:
Command failed with exit code 32: mount -o vers=3 -t nfs 172.23.1.20:/VM/backups /run/xo-server/mounts/7f13127e-3167-4cc1-921a-6dd955984899 mount.nfs: access denied by server while mounting 172.23.1.20:/VM/backups
I can't seem to figure out what I'm doing wrong here. This is my first experience with Xen and everything else seemed to go well...even passing through disks to FreeNAS. This seems pretty straightforward but I just can seem to get it working. Any help would be appreciated. TIA
