VM autostart stopped working

Octopuss

I have no idea what happened. I simply restarted my home server after some time and two out of the four VMs simply didn't start despite being configured to do so in XO-CE.
I googled up some thread where someone suggested flipping the autostart setting off and on via the console, which I did, but the VMS still refuse to autostart. They do start manually just fine.

I have also set every VM's autostart to disabled and then disabled the pool-wide autostart setting, rebooted the server, re-enabled everything, rebooted, and it's still not working
Can anyone tell me what to do here, what to check? I don't really understand this system or Linux in general, everything I do is by copying what I find on internet or by trial and error.
I literally changed nothing about the server and it's just not working all of a sudden

Octopuss

I removed the start-delay parameter from one of the VMs (only this one has it) and now it works again.
I don't get it.
No idea how to achieve a startup delay for a specific VM though. There's also "auto_poweron_delay" and I don't understand what the difference from "start-delay" is.

poddingue

Glad the start-delay removal got that VM going again. I checked and it's a real, documented XCP-ng parameter, "the delay to wait before a call to start up the VM returns", so that explains the behavior.
I couldn't find auto_poweron_delay documented anywhere in the XCP-ng or XO docs, so I'm not sure if it's something XO's advanced VM settings expose under a different name, or something unrelated you picked up while googling.
If you check the VM's Advanced tab in XO and see what's actually set there, that would help pin down which setting you're really dealing with.
Either way this looks like a gap in the autostart guide, since it doesn't mention that start-delay can interact with autostart at all.

Octopuss

@poddingue I found it in a google search.
Supposedly
xe vm-param-set other-config:auto_poweron_delay=<Delay_In_Seconds> uuid=<VM_UUID>

I have no idea whether this is AI hallucinating or whether this was relevant to older releases or what.

But when you wrote it explained the behaviour what did you mean? I don't understand. The setting doesn't delay the start of the VM I am setting it on? Because from what I could see it seemed like it delayed the start of the next VM instead.

poddingue

My earlier wording was the problem here, not your reading of it.

There are two official descriptions and they do not say the same thing. The CLI reference calls start-delay "the delay to wait before a call to start up the VM returns", which is the one I quoted. The XenAPI field description calls it "the delay to wait before proceeding to the next order in the startup sequence". That second one is exactly what you saw: the delay lands on the next VM to start, not on the one you set it on. I quoted the confusing one and then said it explained your behaviour, which it does not really.

What I still cannot explain is why a start-delay would stop a VM autostarting altogether. Neither wording predicts that, so I would rather say I do not know than invent a reason.

On auto_poweron_delay, I went looking and could not find it anywhere in the XCP-ng or XO docs, so I think your instinct about that search result was right.

I have updated the docs PR I had open so it leads with the XenAPI wording and cites both, since the CLI reference phrasing is what sent me wrong in the first place.

Octopuss

@poddingue said:

That second one is exactly what you saw: the delay lands on the next VM to start, not on the one you set it on. I quoted the confusing one and then said it explained your behaviour, which it does not really.

I am really puzzled by this because it makes absolutely no sense whatsoever. I mean why would you design it like that? When I open a VM's settings and find start delay in there, it makes perfect sense to assume it delays start of the one I am messing around with. It makes even less sense for the feature to even exist in the first place when you consider there is no way to set the order in which VMs start.

I am surprised noone ever brought this up and it wasn't changed. It's pretty basic functionality I took for granted when I moved over from ESXi. For example, I have TrueNAS VM and also a torrent seedbox, and the seedbox absolutely has to start after the NAS has fully booted up, otherwise the share it connects to is unavailable. Well it just doesn't work here and everytime I reboot the server (thankfully that doesn't happen often) I have to remember to restart the seedbox VM manually once.

poddingue

I put this on a spare host rather than guess at it twice.

The delay runs after the start, not before it. With start-delay=20 the VM was up in about a second and the call then sat there for the rest of the twenty before returning. So the VM you set it on is already running, and what waits is whatever was going to happen next.

Ordering does exist, but it belongs to vApps: an appliance groups VMs and order decides which goes first (https://docs.xcp-ng.org/vms/vm-lifecycle#vapps). The part that surprised me is that nothing starts an appliance when the host boots. I put both test VMs in one, rebooted, and they stayed down. Autostart is a single bulk start of everything flagged for auto power on, it walks them one at a time, and it ignores order. I set order=2 on a VM and it still went first.

So a delay does hold up the VMs behind it at boot, which is probably what you were seeing, but you cannot pick which one goes first. For your NAS and seedbox I do not think there is a clean answer today. There is an open request for exactly that, a VM that waits on another VM:
https://github.com/vatesfr/xen-orchestra/issues/8433, and your case would be a good one to add to it. What I still cannot explain is why yours stopped starting at all. A delay stalling the ones behind it would explain a late start, not one that never comes up.