Posts made by CodeMercenary

I want to make sure it's not my expectations that are off.

I'm running XO from source and it's up to date, currently commit 0603b.

I came in on a Sunday because there were 44 patches released for xcp-ng. I did the pool update so it updated all three hosts in my pool (Pool Master, Server A and Server B). Then I shut down all the VMs except the one VM running XO. The VM running XO is not on the Pool Master, it's on Server B. I do not have shared storage so I did not want to do a rolling pool reboot.

I rebooted Pool Master. When it came back up and the VMs it ran started to come up, I rebooted Server A. Then I switched to an XO instance (also from source and also current) that isn't running on any xcp host and I used it to shut down the XO instance running on Server B. Soon after I did that, two of the VMs on Server B booted back up. Both of those VMs are set to auto-start.

First question: Did those VMs likely start because the pool master had restarted so it was dutifully restarting the auto-start VMs that weren't running? Or maybe the pool master reboot caused Server B's toolstack to restart which auto-started the VMs?

Then I shut those auto-started VMs down again then waited for the XO instance on Server B to shut down, it was taking quite a while for some reason. Then using the separate XO instance I rebooted Server B.

After Server B booted up, none of the three VMs with auto-start set started automatically. I thought maybe there was a delay for some reason so I waited about 5 minutes but those VMs didn't start.

Second question: Is that expected behavior because the pool master is really in charge of restarting VMs or was it because I had manually shut them down before the reboot so they were trying to maintain that same state?

In retrospect I think I should have used the non-xcp-hosted instance of XO to do all of this stuff so I could have shut down all the VMs then performed a pool reboot without it trying to manually migrate a VM between machines. The all three hosts would have rebooted at the same time and maybe that would have been better.

I have a fairly small infrastructure so I manage that kind of stuff manually and a migration for the remaining VM would have taken a long time. I think this is only my second or third time to patch xcp so I'm still learning the best way to handle it.

@rtjdamen I have become suspicious that my backups might not be as messed up as I thought. Yesterday I noticed that my backups were again in started state long after they would normally be completed. Because some of those backups go to the UNRAID server I mentioned, I decided to do some digging. That UNRAID server does have incoming network activity so I became suspicious that the backups are working just very slow. I checked it again this morning and one of the backups completed after 16 hours, the other one completed after 21 hours. In this case I think the issue is that UNRAID is using the 1Gb adapter instead of the 10Gb adapter.

In the future I'm going to be more careful about deciding that a backup is stuck. I'd like to figure out if there's a way to get more insight into what is happening in the backup, like an ongoing percentage complete and a data transfer speed or total. Would be nice not to have to look at the receiving side for traffic and assume that's the backup, plus some of my backup targets aren't as easy to tell the incoming traffic.

Now I have to dare rebooting the UNRAID server again to see if I can get it to use the right network connection. It must have gotten out of whack when I reset the BIOS and I need to get it back in whack.

@rtjdamen Good to know, thank you. I'm not likely to try to do that without fully understanding what I'm doing. I will just continue to reboot the host if this happens again. It did happen again a week or two ago. My infrastructure isn't so complex that it's impossible to reboot the host, it's just annoying because I have to stay late so nobody is using the servers and I've had servers that failed to boot up after a restart that should have been trivial so I'm always a bit nervous. That was years ago and not when running XCP-ng but it left an emotional scar. I just had the same thing happen with my UNRAID server last Friday, had to clear the BIOS settings to get it to boot again.

@andrewperry Sorry for the delay in responding; I think you posted while I was on vacation. I ended up rebooting the host and have not had the problem return since. Uh oh, I hope I didn't just jinx myself.

@gskger Great to know, thank you.

I actually already have Ollama installed and running with Open WebUI so I can ask it ahead of time to see what it would be like. I've installed some more code specific models that are better suited to that kind of question.

Running the case fans full blast all the time would be a non-starter so I'm glad you let me know that as I count the costs of attempting this.

@gskger Thanks for the input. I also have dual 750W power supplies and I'd be totally fine limiting the power of the card. I'm not looking for crazy performance, just better than what I get with CPUs and enabling things that are a lot harder to even get working with CPU only, like stable diffusion.

I will absolutely trade performance for less noise and heat in my small server closet. Ideally I'd want the GPU card to consume nothing and need very minimal cooling if I'm not actively running a task in ollama. I can currently hear the fans spool up in that server every time I give ollama a task to run. I don't mind the extra noise when doing something as long as it doesn't permanently make the server more noisy.

I do have concerns about possibly needing external cooling for it as suggested in the article, I'd love to not need active cooling. If limiting the power consumption means that the existing case airflow is sufficient then I'd be happy with that.

I built an AI vm a little while ago running ollama on Ubuntu 24.04. I have no GPU in that host so I threw 32GB RAM and 32 cores at it and it works, slowly. I knew that a GPU is really the way to go, especially if I want to be able to do image generation. Then today I saw this article. https://blog.briancmoses.com/2024/09/self-hosting-ai-with-spare-parts.html

A GPU for around $90 could be interesting. Any of you guys have a guess at the difficulty of getting an NVIDIA Tesla M40 (https://www.ebay.com/itm/274978990880) to work in an Ubuntu VM in a PowerEdge 730xd host? I see that the card has power requirements but don't know if the supplies in the server take care of that since this card is 10 years old and the server likely is too, and the card is designed for server use - maybe it wouldn't need tweaking.

Am I insane to be considering this?

@julien-f Thank you, that's super helpful and even easier than I thought it would be.

I guess I'll partially answer my own question by pointing out that I can background the commands with & so it should run them all at once anyway. Sorry, still new to bash scripting and my Linux skills are a bit rusty but improving.

Finishing up my NUT scripting in an XO VM. When NUT sends lowbatt, shutdowncritical, or powerdown I'm going to pull the list of host uuids and then run xo-cli emergencyShutdownHost host=$uuid for each of them.

The complication is that one of these hosts holds the very VM that is running NUT. I'm wondering if xo-cli returns control right away or if it waits for any significant time. What I obviously want to avoid is that the first host in the list is the one with XO+NUT on it and the VM gets suspended before telling the other hosts to shut down.

What's would add insult to injury is that likely when that XO VM resumes after power is restored, it would happily continue its work and ask the other hosts to emergency shutdown.

I can certainly alter the script to carve out the current host and shut it down last, that's probably a good idea regardless, but if the emergency shutdown exits really quickly then I might be able to finish in time. I only have 3 hosts.

Granted, I could put a VM on each of the hosts and have them all using the NUT monitor so I'd have redundancy but I'm trying to make this as simple as possible without making it so simple that it fails me in a power outage.

I noticed, for instance, that when using xo-cli vm.suspend, it waits until the suspend is done so it can return true. I want to make sure the same isn't true of host shutdown.

Obviously I can test this to find out but I'd rather not emergency shutdown one of my hosts during a workday, nor do it from home when I'd have to come back and turn it on. Some evening I can stay late and experiment with it but not this evening or this weekend so I thought I'd ask.

I'm looking for a way to turn off blocking VM suspend without blocking VM stop. I see from xo-cli --list-commands that there is a vm.set described as:

vm.set id=<string> [blockedOperations=<object>]

I have, so far, been unable to figure out what to pass to blockedOperations to have it change a value. Not sure what it means by <object>.

Passing a json string to it doesn't seem to work. I get an error that it must be an object.

I believe I was able to blow away all blocked operations by passing nothing as that arg but I don't want to remove all of it, just the suspend. I realize that when wanting to block stop it makes sense to also block suspend because they, in essence, have the same result. I was hoping to extract an existing object from the vm properties, turn off suspend, then pass that object back to vm.set. Is my quest hopeless?

Interesting to know that v3 seems to be more reliable that v4. I had repeated problems with using NFS for a backup remote and those problems only went away when I changed the remotes to use SMB. I know NFS would be better to use but a backup that happens through an inferior protocol is way better than one that fails using a better protocol.

Maybe I should give NFS another chance but force it to use v3.

In my case, I'd get backups working on NFS and then several days later a backup would fail. Then backups fail every day until I intervene, usually by rebooting the XO VM. Sometimes I'd then have to do cleanup, like releasing a VDI or something. Then it may or may not start working again but if it did start working I'd have another failure a few days later. It's been 2.5 weeks since I switched it to SMB and have had no failures. That's definitely the longest I've gone without a failure from a delta backup to a networked drive.

Note, I also have backups going to a local drive mounted in XO so with all those remote failures I always had a clean backup somewhere. This was in the process of trying to decide if I could trust sending delta backups to a network remote rather than using full backups to a local remote. My initial feelings were that the delta backups didn't work reliably but now I believe the issue was with NFS, not with deltas specifically.

A bit after this I started having trouble again. I scrubbed my delta backup sets, switched to using SMB to access the same share and so far I'm a week in with no trouble at all. Remains to be seen if the strange failures will crop up again but so far I think this is the longest I've gone with the delta backups not having some random failure. Usually, it was a backup getting stuck in Started status for a day or two then switching to Interrupted because I'd reboot the XO VM to unstick it, then one of the VM backups will fail because a VDI is attached to DOM0. I know theoretically NFS is better than SMB but for now the one that breaks least often is the better option. Of course, maybe my issue had nothing to do with NFS but for now it's felt like SMB is more reliable.

So far it seems like there were two things that fixed this.

Setting this in the XO NFS options for the two remotes: soft,timeo=15,retrans=4

Adding vers=4 to the XO NFS option for the Synology connection, as opposed to vers=4.2 or vers=3. It seems the Synology supports version 2, 3 and 4 but not 4.1 or 4.2. So far, version 4 seems to be working better than 3 did.

I had backups fail earlier this week. After more than 24 hours they were still in the started state. I restarted the XO (from source) VM and the backups are listed as Interrupted. I wasn't surprised by this. Since then two of my VMs fail the backup with VDI must be free or attached to exactly one VM. I was also not terribly surprised by that.

Looking at the health info I can see that the drives for those two VMs are listed under VDIs attached to Control Domain. Since those backups failed the other day I know it's safe to forget them. I click the forget button and they remain. I check the logs and the log says:
OPERATION_NOT_ALLOWED(VBD '<guid>' still attached to '<guid for id 0>')

SSH into the host and run list_domains and that's how I confirmed that the second guid is for id 0 which I assume is the control domain since I see it frequently referred to as dom0, right?

That means I need to detach/forget it from dom0 but I can't because it's attached to dom0. At least that's what it feels like is happening.

I tried creating a snapshot for one of the VMs then deleting it to see if that would fix it. I had seen in another forum post that someone fixed it that way. Didn't work.

Must I reboot the host to fix this? This is my primary host and I don't have shared storage so that's kinda a pain. I'll do it if that'll fix my backups but I don't want to do that if there's another reasonable way to fix it without such extreme measures.

@Forza said in All NFS remotes started to timeout during backup but worked fine a few days ago:

I would also pay attention to the async export option as this can lead to corruption if your nfs server crashes or looses power (unless you have battery backups, etc).

I do have battery backup for both NFS hosts. async must be a default since I didn't set it. I have no problem disabling that if it will improve reliability. If it makes a difference, both systems are sharing folders on HDD, not SSDs.

Seems the Synology does support v4. The other day I think I tried v4.2 and it failed so I dropped back to v3 and that worked. I hadn't tried v4.1 and v4. Maybe it'll work better with v4 than it has with v3.

@Forza Sorry, you were correct, I just mixed in another new issue. NFS is currently used only for backups. All my SRs are in local storage. It just happened that I now have backups failing not just because of the NFS issue but because of the VDI issue but I think it's a side-effect of the NFS problem causing the backup to get interrupted so now the VDI is stuck attached to dom0. I should have made that more clear or never mentioned the VDI issue at all.

@Forza Seems you are correct about showmount. On UNRAID running v4 showmount says there are no clients connected. I previously assumed that meant XO only connected during the backup. When I look at /proc/fs/nfsd/clients I see the connections.

On Synology, running v3, showmount does show the XO IP connected. Synology is supposed to support v4 and I have the share set to allow v4 but XO had trouble connecting that way. Synology is pretty limited in what options it lets me set for NFS shares.

Synology doesn't have rcpdebug available. I'll see if I can figure out how to get more logging info about NFS.

I used exportfs -v on UNRAID to look at the share options, without changing anything first, and they were set to:
(async,wdelay,hide,no_subtree_check,fsid=101,sec=sys,rw,secure,root_squash,no_all_squash)

So, fsid was already being used.

The UNRAID remote has worked the last couple days, since I set soft,timeo=15,retrans=4. I'm curious to see how it does over the weekend.

I'm having trouble getting the VDIs detached from dom0.