Posts made by fohdeesha

@cbaguzman hi, you could try a hard mount instead of soft, then it should constantly retry until it sees the mount again https://www.ibm.com/docs/en/spectrum-protect/8.1.6?topic=considerations-nfs-hard-soft-mounts

@olivierlambert like I said in my previous reply I believe this would take quite a bit of rewriting in XAPI to accommodate pools with mismatched hardware. The interface rename tool might work in the meantime? https://xcp-ng.org/docs/networking.html#renaming-nics

@mike hmm, NFS locking is handled by statd so make sure that is running properly on your NFS server.

Also, this may be related to the .nfsXXXX files that are generated/used by the system to keep track of open files that should later be deleted when closed. These things:

root@zfs[/mnt/chunker/xen-backup/XH-01]# ls -lh
total 18
-rw-r--r--   1 xen  wheel     0B Oct 19 05:24 .nfs000000000000020300000002
drwx------   3 xen  wheel     3B Jun 29 05:59 xo-config-backups
drwx------   3 xen  wheel     3B Jun 29 05:59 xo-pool-metadata-backups
drwxr-xr-x  15 xen  wheel    15B Sep 26 09:39 xo-vm-backups

In your backup of the backup job, can you have it exclude any .nfs* files so it doesn't try to access or copy them, and see if the locking errors go away?

@dr4s1l-0 @Dr4s1l-0 you will not see the config drive mounted in your system, cloud-init only mounts it temporarily very early on in the boot process, copies the found config files off of it, and unmounts it. Mounting it manually will just cause more issues. You can probably see it being mounted early on in the logs in /var/log/cloud-init.log

Cloud-init has introduced a ton of bugs this year and has no documentation regarding them, it has been a nightmare to try and track down these issues - I have a support customer with the same issue on recent versions of ubuntu and I've gotten nowhere. At this point I think it's best to take it up with the upstream project, cloud-init - from the XOA/XCP perspective we're doing everything we can, resulting in presenting a properly configured config drive - any issue after that is the cloud-init package

@encryptblockr Hi! by cloud image I'm assuming you mean cloud-init? Cloud-init has some issues and their documentation hasn't been updated to cover a lot of changes so I've been dealing with debugging it for a customer on ubuntu, but it's mostly related to network config. Your displayed config should work no problem for a cloud template, the important thing is that the linux/OS partition is at the end of the disk, after all the other partitions - this way cloud-init can expand it without running over partitions after it. Your fdisk output shows that it's the last partition, so you should be good.

I just confirmed with tests that Disaster Recovery jobs do indeed set the blocked-operations parameter for backed up VMs. As Julien said, did you ever force-start this VM? That would have removed start from the blocked-operations field, which is why we don't see it in your param-list, and why it's now being allowed to autostart.

@thorcraftit seems like perhaps a permissions error, what does your share properties on truenas look like for this host? Is the allowed IP or denied IP boxes populated at all? Attach a screenshot if you can. Here's what mine looks like:

Note that the mapall user and group must have permissions on the shared folder, I typically create a user on the freenas system just for xen shares, and give it ownership of the folder/dataset that's being shared. this ensures you don't have any weird permissions issues. I think in your case the key boxes to examine are the authorized networks however.

What does your Services > NFS > Settings look like? here's mine:

Lastly and perhaps most importantly, can you run "yum install nmap" and then "nmap 172.16.60.200" on both a working host, and then this non-working host? Paste the results of both here - if they differ, for instance if the non-working server doesn't see any RPC services running like I suspect, that will heavily point towards a firewall/network/ACL issue

@station-already hehe glad you got it figured out Indeed, management interface changes are pretty "violent" in terms of how much they change and interrupt, so a reboot helps quite often - same for changing the virtual adapter/PIF assigned to a VM - alway a good idea to reboot the VM

@olivierlambert indeed, I believe it's just:

xe host-management-reconfigure pif-uuid=<PIF_UUID>

with the PIF_UUID being the UUID of the new PIF interface (bond, vlan, etc) you would like to move the existing management interface to

@station-already Hi! This can be done in XOA, but it requires deleting the existing management interface, then creating a new one on the new PIF you've created (a vlan 10 on eth0 PIF for example). You say you can't lose access to the host however, so that is probably not an option. XCP-ng center, the windows client, has a bit of a workaround, it has a dropdown box that will allow you to change the PIF of an existing management interface/IP address to a new interface. In XCP-ng center just go to the Networking tab of the host, then hit configure at the bottom beneath the management interfaces section.

It will have a new pop up window with a dropdown for where you want the existing management interface to exist. Assuming you've already created the vlan network you want in xcp-ng, it should show up in this list. just choose it and hit ok:

@garrya Hi, it sounds like you've set up VLAN interfaces correctly on the XCP-ng side, but the switch they are connected to also needs to be aware of VLAN tags to pass them, or else it will only pass untagged traffic as you noticed. You'll need a smart/managed switch with VLAN capability to do so

@slynch OK, that's very good news then! Debian was still having issues where any configuration applied via cloud-init would just be added to the existing default DHCP config, unless you took care to build the VM without the default interfaces config beforehand.

@slynch Interesting! since you left the default DHCP netplan, does the VM also have a DHCP config and grab a DHCP address in addition to your cloud-init config?

Also, you mean when there is netplan content, cloud-init does finally apply netplan apply like it's supposed to? interesting

@slynch that end of the issue sounds like a cloud-init bug/issue, which doesn't surprise me (I've been fighting with them since I started here at Vates). I suppose you could manually work around it, somewhere in the cloud-init options/parameters is optional commands to pass and run in the VM on first run, I suppose you could pass "netplan apply" there.

@s-pam If you don't mind about power usage, high performance (that you've already selected) is the way to go. It will avoid entering C states as often and leave the CPU clocked high among other things so there is no latency for workloads waiting on the CPU existing C states etc.

Leave x2APIC enabled, it helps to distribute interrupts between multiple cores/CPUs. There's no downside to having it enabled unless the hypervisor doesn't support it (and you would know immediately by having errors).

Regarding your last setting, also referred to as CCX as NUMA, this changes how the CPU cores are presented to the hypervisor (either as one NUMDA domain, or multiple, one for each set of cores that share cache). From the benchmarks I can find, leaving this on gains maybe 3 to 4 percent improvement https://blogs.vmware.com/performance/2020/05/app-perf-vmware-vsphere-amd-epyc-rome.html

@slynch Indeed this needs to be passed in the meta-data file, and there's currently no way in XOA in the GUI to pass anything into this file. This thread has some more info and sums up my feelings on cloud-inits documentation state: https://gist.github.com/Informatic/0b6b24374b54d09c77b9d25595cdbd47

We do write out a meta-data file, but as of now there's no way to pass custom contents into it, it is just used to pass the VM UUID into cloud-init: https://github.com/vatesfr/xen-orchestra/blob/master/packages/xo-server/src/xapi/index.js#L2019

Indeed this seems like yet another broadcom driver/firmware issue (not uncommon)

@olivierlambert I believe he means be able to rename / map a hosts physical interfaces to different namings. For instance on host 1, "network 0 / eth0" would be mapped to the physical eth0 interface, but on another host in the pool with a different NIC configuration, he could map the same pool "network 0 / eth0" to the server's physical eth1.

@S-Pam this certainly isn't currently supported, I believe it would take a lot of rewriting the network backend to make it possible.

I think when I last asked @julien-f about this he said it work work (just as you described), but the issue is xo-server will not reload certs without restarting the process. So the next time your let's encrypt instance updates those certs, xo-server will have no idea and you'll need to schedule a restart of that service after certs are updated

Hi, I have never attempted to install guest-utils on FreeNAS (FreeBSD based). Our documentation is for generic FreeBSD installs, however FreeNAS is highly customized and I believe they have most of the repositories disabled (which is why you get the repo not found error). It seems they don't have the required build tools to make either, but I could be wrong.

I'm not sure where you're seeing instructions to do "make install", we have a dedicated document section just for FreeNAS specifically that does not require running make. Try following it here: https://xcp-ng.org/docs/guests.html#freenas-truenas