XCP-ng 7.5.0 final is here

GregW

This post is deleted!

dsiminiuk

@olivierlambert The VDIs are over shared ISCSI on FreeNAS VDEVs. There are other VMs with VDIs on the same RAID10 LUN that are running fine.

It's actually 2 VDIs and I moved both (boot and swap) of the failed VM to an alternate RAID5 LUN and it made no difference.

Is there a log I should be looking in to see more detail?

olivierlambert

Check the /var/log/xensource.log and SMlog about this VDI error.

Again, a PANIC on CPU 0 isn't a good news regarding your hardware.

txsastre

well installed from scratch on my 3 servers
everything fine, but, when trying to activate HA it gives me an error. The same when I upgraded 7.4.1 to 7.5 RC.

3 hosts are connected via HBA to a one Storage on a fibre channel.

tried to connect only 2 hosts with a few combinations (host1 and host2, host1 and host3, host2 and host3), also trying to configure HA with one host (yep little strange) it gave me the same error.

0_1533928426464_Selecció_006.jpg

olivierlambert

Please provide real logs and not just a XCP-ng screenshot

How did you enable HA by the way?

dsiminiuk

The CPU PANIC was on server1 (not pool master) so I've shut it down and I'll do a fresh install there later. It won't even show me the NICs that I know are there. Might be a hardware problem indeed.

I left all VMs powered down and tried to capture SMlog and xensource.log for the duration of the startup failure event.

I see "Raised Server_error(SESSION_INVALID," but have no idea if this is related to the error.
I don't have privs to upload the files.

olivierlambert

Use pastebin or hastebin to put your logs there

SESSION INVALID is not related

dsiminiuk

xensource.log https://cloud.k5rwk.org/index.php/s/BjJBBFxcg9EkknP
SMlog https://cloud.k5rwk.org/index.php/s/YxCYjAKRgcYgmKn

borzel

VDI_MISSING

xensource.log:500

Aug 10 14:58:43 xs2 xenopsd-xc: [error|xs2|15 |Parallel:task=50.atoms=1.(VBD.plug RO vm=67bd9da9-9cb7-1882-9ad4-68e65e6095a2)|xenops] Re-raising exception VDI_MISSING: OpaqueRef:fa60f4c9-7095-35d7-6c91-2436713508a0; OpaqueRef:e08c2398-4e43-4cdd-af5b-41926a5f9d76

dsiminiuk

Well, I detached the VDIs, deleted the VM, created a new one, attached the VDIs and it booted up. This is solved for me, but I don't know the root cause.

borzel

Yehaa!
I hope you had a backup if it had not worked

txsastre

@olivierlambert sorry. Where can i get them from?
I tried to enable HA as always using xcp-ng center.

olivierlambert

Same as @dsiminiuk xensource and SMlog

dsiminiuk

I've installed from CDROM a new clean install of 7.5 and had all my VMs running. I added the 2nd host to the pool and migrated some of the VMs from the pool master to the 2nd server fine. One however, abended with a XenOpts error (don't have the details now). And the VM that was powered off while transferring and now it won't start. Status is "Starting VM..." for 10+ minutes now.
Something's not quite right here.

dsiminiuk

And when I changed the home host from "do not assign" to the pool master, then the VM started up just fine. It's as if it couldn't identify a host to start on and timed out at 20 minutes.
Very weird.

olivierlambert

What did you use to "assign" a host for the VM? Affinity?

onur

Upgrade from 7.4 to 7.5.0 done flawlessly via ISO upgrade. (just saw an Failed to start Load Kernel Modules message after the install, but later it initialize Apply Kernel Variable procesure, I guess it updated the kernel modules)

Did not try the new USB passthrough but the old style still works.
Now eager to test ZFS!

Thank you so much for XCP-ng!

dsiminiuk

@olivierlambert I edited the properties of the VM and changed the "Home Server" property to server1 rather than do not assign. Then it was able to start. When I looked at XCP ng Center this morning the status icon for that VM was yellow. I restarted the tool stack on the server and the VM status turned green. I also received an email from the pool...

Field                Value
-----                -----
Name:                PBD_PLUG_FAILED_ON_SERVER_START
Priority:        3
Class:                Host
Object UUID:        c3970498-bc7f-4a49-8fe2-23e1e6dcb6bb
Timestamp:        20180811T06:27:04Z
Message UUID:        88db4c9a-6435-0bd3-2237-5d4d904a5782
Pool name:        pool1

Maybe related, maybe not. XCP ng Center crashed twice along the way and mentioned to look in the log in D:\Users\<my_user>\AppData\Roaming\XCP-ng\XCP-ng Center\logs\XCP-ng Center.log

Recurring ERRORs

2018-07-12 22:11:48,931 ERROR XenAdmin.Dialogs.LicenseStatus [23] - Couldn't get the server time for pool1 because: Object reference not set to an instance of an object.
2018-07-12 22:11:48,931 ERROR XenAdmin.Utils.AsyncServerTime [23] - Failed to fetch server time for host pool1 because Object reference not set to an instance of an object.

2018-07-12 22:34:27,141 ERROR XenAdmin.Program [Named pipe thread] - Exception in Invoke (ControlType=XenAdmin.MainWindow, MethodName=<ConnectPipe>b__1)
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'MainWindow'.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at XenAdmin.Program.Invoke(Control c, MethodInvoker f)

other error removed, too old to be pertinent to 7.5

Everything appears stable at the moment but I'm hesitant to manipulate my VMs just for the sake of doing so. I've taken fresh backups of all my VMs so I have a restore point. If there are specific tests you would recommend to peel back the onion, I am willing to try.

dsiminiuk

I edited that VM again and removed the startup host preference. Shut it down. Started it up again and it worked. That's a good sign.

dsiminiuk

More info. I made server2 the pool master and it started migrating all the VMs from server1.

When it came to the only WIndows VM I have, I received an error on the migration dialog in XCP ng Center. From the log file...

2018-08-12 14:45:42,302 INFO  Audit [21] - Operation success: DelegatedAsyncAction: pool1: Pool 86708278-0d7d-2bb1-5024-b04c39690e15 (pool1): Completed
2018-08-12 14:45:42,393 WARN  XenAdmin.Network.TaskPoller [72] - Action failed due to API failure:
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at XenAdmin.Network.TaskPoller.poll()
   at XenAdmin.Network.TaskPoller.PollToCompletion()
   at XenAdmin.Actions.EvacuateHostAction.Run()
   at XenAdmin.Actions.AsyncAction.RunWorkerThread(Object o)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
2018-08-12 14:45:42,393 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.destroy'
2018-08-12 14:45:42,434 ERROR XenAdmin.Actions.EvacuateHostAction [72] - There was an exception putting the host OpaqueRef:6d59f477-2e3c-40f1-920b-575719f33cee into maintenance mode.  Removing other_config key.
2018-08-12 14:45:42,434 ERROR XenAdmin.Actions.EvacuateHostAction [72] - Internal error: Xenops_interface.Does_not_exist(_)
Internal error: Xenops_interface.Does_not_exist(_)
2018-08-12 14:45:42,435 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'host.remove_from_other_config'
2018-08-12 14:45:42,475 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'Async.host.enable'
2018-08-12 14:45:42,513 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.remove_from_other_config'
2018-08-12 14:45:42,552 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.add_to_other_config'
2018-08-12 14:45:42,592 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.remove_from_other_config'
2018-08-12 14:45:42,633 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.add_to_other_config'
2018-08-12 14:45:43,765 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.destroy'
2018-08-12 14:45:43,806 ERROR XenAdmin.Actions.AsyncAction [72] - Internal error: Xenops_interface.Does_not_exist(_)
2018-08-12 14:45:43,806 ERROR XenAdmin.Actions.AsyncAction [72] -    at XenAdmin.Network.TaskPoller.poll()
   at XenAdmin.Network.TaskPoller.PollToCompletion()
   at XenAdmin.Actions.EvacuateHostAction.Run()
   at XenAdmin.Actions.AsyncAction.RunWorkerThread(Object o)
2018-08-12 14:45:43,808 WARN  Audit [72] - Operation failure: EvacuateHostAction: pool1: Pool 86708278-0d7d-2bb1-5024-b04c39690e15 (pool1): Host cb05539d-3359-4535-97a9-7f2b635f44fd (xs1): Server 'xs1' entering maintenance mode...

That's the error I see on the dialog box that is doing the "Entering Maintenance mode" work of migrating the VMs.

It happened on the same Windows VM as before and now it's shut down and the maint mode wizard allowed me to continue to migrate the last Linux VM. As soon as the pool master appeared to change I received another error popup box.
There has been an unexpected error. Technical details are in the logfile saved here https://cloud.k5rwk.org/index.php/s/e9weBDcyJmYzQHK

Bottom snip...

2018-08-12 14:50:38,477 ERROR XenAdmin.InvokeHelper [Main program thread] - Exception in Invoke
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at XenAdmin.TabPages.GeneralTabPage.PoolAdditionalLicenseString()
   at XenAdmin.TabPages.GeneralTabPage.generateGeneralBox()
   at XenAdmin.TabPages.GeneralTabPage.BuildList()
   at System.Action.Invoke()
   at XenAdmin.OtherConfigAndTagsWatcher.OnTagsChanged()
   at XenAdmin.OtherConfigAndTagsWatcher.<>c.<connection_ConnectionStateChanged>b__25_0()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at XenAdmin.InvokeHelper.<>c__DisplayClass4_0.<Invoke>b__0()
2018-08-12 14:50:38,480 ERROR XenAdmin.Program [Main program thread] - Exception in Invoke
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at XenAdmin.TabPages.GeneralTabPage.PoolAdditionalLicenseString()
   at XenAdmin.TabPages.GeneralTabPage.generateGeneralBox()
   at XenAdmin.TabPages.GeneralTabPage.BuildList()
   at System.Action.Invoke()
   at XenAdmin.OtherConfigAndTagsWatcher.OnTagsChanged()
   at XenAdmin.OtherConfigAndTagsWatcher.<>c.<connection_ConnectionStateChanged>b__25_0()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at XenAdmin.InvokeHelper.<>c__DisplayClass4_0.<Invoke>b__0()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at XenAdmin.Program.<>c__DisplayClass66_0.<Invoke>b__0()
2018-08-12 14:50:38,481 FATAL XenAdmin.Program [Connection to 192.168.1.15] - Uncaught exception
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at XenAdmin.Program.Invoke(Control c, Delegate f, Object[] p)
   at XenAdmin.MainWindow.System.ComponentModel.ISynchronizeInvoke.Invoke(Delegate method, Object[] args)
   at XenAdmin.OtherConfigAndTagsWatcher.connection_ConnectionStateChanged(Object sender, EventArgs e)
   at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
   at XenAdmin.Network.XenConnection.OnConnectionStateChanged()
   at XenAdmin.Network.XenConnection.EndConnect(Boolean clearCache, ConnectTask task, Boolean exiting)
   at XenAdmin.Network.XenConnection.HandleConnectionLost(ConnectTask task, Pool pool, String poolopaqueref)
   at XenAdmin.Network.XenConnection.HandleConnectionResult(ConnectTask task, Exception error, Pool pool)
   at XenAdmin.Network.XenConnection.ConnectWorkerThread(Object o)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)