@kagbasi-ngc I figured out what the problem was, so wanted to post it for the benefit of the community.

The root cause for me ended up being a firewall issue on the Windows Server where I was running the NFS server role. While I hadn't made any changes to the firewall, the profile of the network interface connected to the management network had changed from Domain to Public/Private, thus, Windows Defender Firewall applied the most restrictive settings since it saw that the NIC wasn't connected to the Domain profile (even though it physically was).

This post on Serverfault (https://serverfault.com/a/647201/189248) clued me in and I looked at the Network Location Awareness service (NLA). It was set automatic and was running, so I restarted it and this reverted the profile of the NIC back to Domain and caused Windows Defender Firewall to apply the previously configured rules.

It is important to note that this situation won't be common, as most of you won't be running your NAS on a dual-homed domain controller that is also running the DHCP server role. Hope this helps someone.

@Prilly Looks like @lawrencesystems has a good explainer video on YouTube at https://youtu.be/weVoKm8kDb4?si=QH93rOKaXLglrIbS

Check it out when you get a chance, thanks.

@stormi Sorry for the delayed reply. I did as you requested, and below are the resulting log files. I started the VM at timestamp Dec 17 14:43:34 and stopped it about 2 minutes later.

• daemon.log
• xensource.log

If I can be of assistance, don't hesitate to ask please. Thank you.

@olivierlambert No, it's the other way around. Once the subject user is a memberof 3 or more groups, the problem occurs.

@manilx Yeah, not doing it in production, just in a test environment. Plus I only did it because I actually found a write-up by the Vates team on how to get it done, which confirmed to me that while they don't recommend it for production, they understand that it has some merit in a testing environment and want to see it working.

That said, I definitely hear your admonition and would never do it in a production setting.

@olivierlambert Certainly strange, then, that the same settings and same plugin works in XOA but not in XOCE. We could argue perhaps some configuration variances within the Domain Controllers, but given that I'm the same person that configured both environments - I'm highly confident that I stood up the DCs in the same way. Only things that would obviously be different would be the Domain Name and OU Structure.

I'll confirm the Forest and Domain functional levels later this evening and perhaps run an XOA trial and see if it works. I'll report back.

@olivierlambert So XOA is running on local storage of host #1 (VMH01) and all the VMs run off an NFS datastore which I've checked and is not in read-only mode or full (has about 60% free space).

What I haven't checked is whether the partition being used as the Local SR on VMH01 is perhaps the culprit. I ran xe check yesterday and the only three things it complained about were NTP, Updates, and not having Internet connectivity.

I should be in the office in about an hour, so I'll inspect and report back.

Good-day Folks,

Today I attempted to move a VDI from one NFS SR to another NFS SR and it failed with the following error:

vdi.migrate
{
  "id": "c6967c64-01a3-4910-9783-78c36629c080",
  "sr_id": "921094bf-12a3-78a2-f470-a2b6ad342647"
}
{
  "code": "VDI_COPY_FAILED",
  "params": [
    "warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 3 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 2 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 1 more time...
Fatal error: exception Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\")
"
  ],
  "task": {
    "uuid": "4c1c4a91-f5a3-8a7d-6b55-e9bbca5f8fb0",
    "name_label": "Async.VDI.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250103T13:40:48Z",
    "finished": "20250103T13:41:06Z",
    "status": "failure",
    "resident_on": "OpaqueRef:e11885bb-3b4d-41f1-3db9-688b9fbc71ae",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VDI_COPY_FAILED",
      "warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 3 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 2 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 1 more time...
Fatal error: exception Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\")
"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 11330))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 146))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
  },
  "message": "VDI_COPY_FAILED(warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 3 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 2 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 1 more time...
Fatal error: exception Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\")
)",
  "name": "XapiError",
  "stack": "XapiError: VDI_COPY_FAILED(warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 3 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 2 more times...
warning: caught Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\"); will retry 1 more time...
Fatal error: exception Failure(\"Failed to find parent (tried [  ] with search_path /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a/.; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /dev/mapper; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a; /var/run/sr-mount/30d873a3-3523-1952-e923-bcaa02b9255a)\")
)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1076:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1110:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1100:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1273:14)"
}

Is this 30d873a3-3523-1952-e923-bcaa02b9255a the UUID of the parent being referenced in the error message? If so, I can manually browse through the NFS path and see if I can find it. That said, I can confirm that both SRs are online and working, and I have completed other VDI migrations without much issue.

@olivierlambert Thanks, for at least acknowledging that I'm not crazy. I'm willing to help out with testing, just let me know what's needed of me. I'm not a developer by any stretch of the imagination, but I am a seasoned SysAdmin with just enough Linux knowledge to be a little dangerous...lol.

@Danp Confirmed - it was the Windows Firewall issue again. I applied the fix from the last time it happened, and things are working again.

For anyone interested - the solution is here: https://xcp-ng.org/forum/post/86828

@olivierlambert This issue highlighted to me how vague that "NO HOSTS AVAILABLE" error message is in XO. Any chance this could be improved upon?

@Danp Yes sir, I verified that all infrastructure services (including networking was up) were up and functional.

I log into XOA with AD Integration, so the fact that I could login, the SMB SRs being available (pointing to the same server that hosts the NFS Server role, by the way), and being able to access the same NFS exports from another physical Linux host, made me think that whatever is happening must be local to all three hosts.

I'm home now, but once the kids go to sleep I'll run out to the lab and run those commands and report the output. Thanks.

UPDATE:
Now that I think of it, I wonder if this same issue that affected me in this thread has read its head again. I'll check when I go in.

Good-day Folks,

Happy New Year to you all. So, today I came back to my air-gapped test lab (after about a week away), to find all systems in the lab without power. So I gradually powered everything up, including my 3-node XCP-ng pool. When HOST #1 came up and I confirmed that XOA had come up, I logged into it then brought up the remaining two hosts.

Once all three hosts were up, I attempted to bring up the VMs (after observing that they did not power up automatically, as they should have). That's when I noticed the "NO HOSTS AVAILABLE" error in XOA. Every VM I attempted to start, would fail with that error. After checking over the ADVANCED tab of all the VMs I was looking to start and not noticing anything out of the ordinary, I turned my attention to the STORAGE REPOSITORIES. That's when I immediately noticed that none of my NFS SRs have reconnected. This now explained why the VMs won't boot; their VDIs existed on the NFS SRs.

So I went to the respective NFS SR and clicked on the "Reconnect to All Hosts" button, and was met with the error: POST_ATTACH_SCAN_FAILED. I then went to each host, from within XOA and clicked on the "Connect" button - that's when I saw a different error message popup: "NFS Service Not Detected on Host".

Has anyone seen this before?

I've already rebooted all hosts, to no avail. And I can confirm that the NFS server is up, running, and the NFS exports are available and can be accessed from other systems within the lab environment.

@stormi Yes it does. I just deleted the VTPM and tried starting it, but no success. Still the same issue - can't initialize the display.

I also disabled Secure Boot, and got the same results.

@stormi No worries at all, I understand.

@stormi Sorry for the delayed reply. I did as you requested, and below are the resulting log files. I started the VM at timestamp Dec 17 14:43:34 and stopped it about 2 minutes later.

• daemon.log
• xensource.log

If I can be of assistance, don't hesitate to ask please. Thank you.

@pdonias As requested, I added the log filter to the XO config file, restarted the xo-server service, and then ran the LDAP tester from the UI twice. Afterwards, I then ran journalctl -u xo-server, and this is all I see in the logs:

ON XOA:

Dec 16 17:15:59 xoa xo-server[418777]: 2024-12-16T22:15:59.962Z xo:api WARN user | plugin.test(...) [165ms] =!> Error: could not authenticate user
Dec 16 17:17:46 xoa xo-server[418777]: 2024-12-16T22:17:46.904Z xo:api WARN user | plugin.test(...) [13ms] =!> Error: could not authenticate user

ON XOCE:

Dec 16 17:22:34 XO1 xo-server[85750]: 2024-12-16T22:22:34.618Z xo:api WARN user | plugin.test(...) [62ms] =!> Error: could not authenticate user
Dec 16 17:23:04 XO1 xo-server[85750]: 2024-12-16T22:23:04.792Z xo:api WARN user | plugin.test(...) [11ms] =!> Error: could not authenticate user

I am still able to login, if I reduce the security groups I'm member of to 2 or less. 3 or more, results in failure.

@pdonias Will do as requested and report back. I have a meeting to get to now, but should be able to do this afterwards.

@olivierlambert As promised, I checked my user account at work and it is a member of 5 security groups, and the primary group is not Domain Users.