XCP-ng 7.5.0 final is here

olivierlambert

Use pastebin or hastebin to put your logs there

SESSION INVALID is not related

dsiminiuk

xensource.log https://cloud.k5rwk.org/index.php/s/BjJBBFxcg9EkknP
SMlog https://cloud.k5rwk.org/index.php/s/YxCYjAKRgcYgmKn

borzel

VDI_MISSING

xensource.log:500

Aug 10 14:58:43 xs2 xenopsd-xc: [error|xs2|15 |Parallel:task=50.atoms=1.(VBD.plug RO vm=67bd9da9-9cb7-1882-9ad4-68e65e6095a2)|xenops] Re-raising exception VDI_MISSING: OpaqueRef:fa60f4c9-7095-35d7-6c91-2436713508a0; OpaqueRef:e08c2398-4e43-4cdd-af5b-41926a5f9d76

dsiminiuk

Well, I detached the VDIs, deleted the VM, created a new one, attached the VDIs and it booted up. This is solved for me, but I don't know the root cause.

borzel

Yehaa!
I hope you had a backup if it had not worked

txsastre

@olivierlambert sorry. Where can i get them from?
I tried to enable HA as always using xcp-ng center.

olivierlambert

Same as @dsiminiuk xensource and SMlog

dsiminiuk

I've installed from CDROM a new clean install of 7.5 and had all my VMs running. I added the 2nd host to the pool and migrated some of the VMs from the pool master to the 2nd server fine. One however, abended with a XenOpts error (don't have the details now). And the VM that was powered off while transferring and now it won't start. Status is "Starting VM..." for 10+ minutes now.
Something's not quite right here.

dsiminiuk

And when I changed the home host from "do not assign" to the pool master, then the VM started up just fine. It's as if it couldn't identify a host to start on and timed out at 20 minutes.
Very weird.

olivierlambert

What did you use to "assign" a host for the VM? Affinity?

onur

Upgrade from 7.4 to 7.5.0 done flawlessly via ISO upgrade. (just saw an Failed to start Load Kernel Modules message after the install, but later it initialize Apply Kernel Variable procesure, I guess it updated the kernel modules)

Did not try the new USB passthrough but the old style still works.
Now eager to test ZFS!

Thank you so much for XCP-ng!

dsiminiuk

@olivierlambert I edited the properties of the VM and changed the "Home Server" property to server1 rather than do not assign. Then it was able to start. When I looked at XCP ng Center this morning the status icon for that VM was yellow. I restarted the tool stack on the server and the VM status turned green. I also received an email from the pool...

Field                Value
-----                -----
Name:                PBD_PLUG_FAILED_ON_SERVER_START
Priority:        3
Class:                Host
Object UUID:        c3970498-bc7f-4a49-8fe2-23e1e6dcb6bb
Timestamp:        20180811T06:27:04Z
Message UUID:        88db4c9a-6435-0bd3-2237-5d4d904a5782
Pool name:        pool1

Maybe related, maybe not. XCP ng Center crashed twice along the way and mentioned to look in the log in D:\Users\<my_user>\AppData\Roaming\XCP-ng\XCP-ng Center\logs\XCP-ng Center.log

Recurring ERRORs

2018-07-12 22:11:48,931 ERROR XenAdmin.Dialogs.LicenseStatus [23] - Couldn't get the server time for pool1 because: Object reference not set to an instance of an object.
2018-07-12 22:11:48,931 ERROR XenAdmin.Utils.AsyncServerTime [23] - Failed to fetch server time for host pool1 because Object reference not set to an instance of an object.

2018-07-12 22:34:27,141 ERROR XenAdmin.Program [Named pipe thread] - Exception in Invoke (ControlType=XenAdmin.MainWindow, MethodName=<ConnectPipe>b__1)
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'MainWindow'.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at XenAdmin.Program.Invoke(Control c, MethodInvoker f)

other error removed, too old to be pertinent to 7.5

Everything appears stable at the moment but I'm hesitant to manipulate my VMs just for the sake of doing so. I've taken fresh backups of all my VMs so I have a restore point. If there are specific tests you would recommend to peel back the onion, I am willing to try.

dsiminiuk

I edited that VM again and removed the startup host preference. Shut it down. Started it up again and it worked. That's a good sign.

dsiminiuk

More info. I made server2 the pool master and it started migrating all the VMs from server1.

When it came to the only WIndows VM I have, I received an error on the migration dialog in XCP ng Center. From the log file...

2018-08-12 14:45:42,302 INFO  Audit [21] - Operation success: DelegatedAsyncAction: pool1: Pool 86708278-0d7d-2bb1-5024-b04c39690e15 (pool1): Completed
2018-08-12 14:45:42,393 WARN  XenAdmin.Network.TaskPoller [72] - Action failed due to API failure:
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at XenAdmin.Network.TaskPoller.poll()
   at XenAdmin.Network.TaskPoller.PollToCompletion()
   at XenAdmin.Actions.EvacuateHostAction.Run()
   at XenAdmin.Actions.AsyncAction.RunWorkerThread(Object o)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
2018-08-12 14:45:42,393 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.destroy'
2018-08-12 14:45:42,434 ERROR XenAdmin.Actions.EvacuateHostAction [72] - There was an exception putting the host OpaqueRef:6d59f477-2e3c-40f1-920b-575719f33cee into maintenance mode.  Removing other_config key.
2018-08-12 14:45:42,434 ERROR XenAdmin.Actions.EvacuateHostAction [72] - Internal error: Xenops_interface.Does_not_exist(_)
Internal error: Xenops_interface.Does_not_exist(_)
2018-08-12 14:45:42,435 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'host.remove_from_other_config'
2018-08-12 14:45:42,475 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'Async.host.enable'
2018-08-12 14:45:42,513 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.remove_from_other_config'
2018-08-12 14:45:42,552 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.add_to_other_config'
2018-08-12 14:45:42,592 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.remove_from_other_config'
2018-08-12 14:45:42,633 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.add_to_other_config'
2018-08-12 14:45:43,765 INFO  XenAPI.Session [72] - Invoking JSON-RPC method 'task.destroy'
2018-08-12 14:45:43,806 ERROR XenAdmin.Actions.AsyncAction [72] - Internal error: Xenops_interface.Does_not_exist(_)
2018-08-12 14:45:43,806 ERROR XenAdmin.Actions.AsyncAction [72] -    at XenAdmin.Network.TaskPoller.poll()
   at XenAdmin.Network.TaskPoller.PollToCompletion()
   at XenAdmin.Actions.EvacuateHostAction.Run()
   at XenAdmin.Actions.AsyncAction.RunWorkerThread(Object o)
2018-08-12 14:45:43,808 WARN  Audit [72] - Operation failure: EvacuateHostAction: pool1: Pool 86708278-0d7d-2bb1-5024-b04c39690e15 (pool1): Host cb05539d-3359-4535-97a9-7f2b635f44fd (xs1): Server 'xs1' entering maintenance mode...

That's the error I see on the dialog box that is doing the "Entering Maintenance mode" work of migrating the VMs.

It happened on the same Windows VM as before and now it's shut down and the maint mode wizard allowed me to continue to migrate the last Linux VM. As soon as the pool master appeared to change I received another error popup box.
There has been an unexpected error. Technical details are in the logfile saved here https://cloud.k5rwk.org/index.php/s/e9weBDcyJmYzQHK

Bottom snip...

2018-08-12 14:50:38,477 ERROR XenAdmin.InvokeHelper [Main program thread] - Exception in Invoke
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at XenAdmin.TabPages.GeneralTabPage.PoolAdditionalLicenseString()
   at XenAdmin.TabPages.GeneralTabPage.generateGeneralBox()
   at XenAdmin.TabPages.GeneralTabPage.BuildList()
   at System.Action.Invoke()
   at XenAdmin.OtherConfigAndTagsWatcher.OnTagsChanged()
   at XenAdmin.OtherConfigAndTagsWatcher.<>c.<connection_ConnectionStateChanged>b__25_0()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at XenAdmin.InvokeHelper.<>c__DisplayClass4_0.<Invoke>b__0()
2018-08-12 14:50:38,480 ERROR XenAdmin.Program [Main program thread] - Exception in Invoke
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at XenAdmin.TabPages.GeneralTabPage.PoolAdditionalLicenseString()
   at XenAdmin.TabPages.GeneralTabPage.generateGeneralBox()
   at XenAdmin.TabPages.GeneralTabPage.BuildList()
   at System.Action.Invoke()
   at XenAdmin.OtherConfigAndTagsWatcher.OnTagsChanged()
   at XenAdmin.OtherConfigAndTagsWatcher.<>c.<connection_ConnectionStateChanged>b__25_0()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at XenAdmin.InvokeHelper.<>c__DisplayClass4_0.<Invoke>b__0()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at XenAdmin.Program.<>c__DisplayClass66_0.<Invoke>b__0()
2018-08-12 14:50:38,481 FATAL XenAdmin.Program [Connection to 192.168.1.15] - Uncaught exception
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
   at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
   at XenAdmin.Program.Invoke(Control c, Delegate f, Object[] p)
   at XenAdmin.MainWindow.System.ComponentModel.ISynchronizeInvoke.Invoke(Delegate method, Object[] args)
   at XenAdmin.OtherConfigAndTagsWatcher.connection_ConnectionStateChanged(Object sender, EventArgs e)
   at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
   at XenAdmin.Network.XenConnection.OnConnectionStateChanged()
   at XenAdmin.Network.XenConnection.EndConnect(Boolean clearCache, ConnectTask task, Boolean exiting)
   at XenAdmin.Network.XenConnection.HandleConnectionLost(ConnectTask task, Pool pool, String poolopaqueref)
   at XenAdmin.Network.XenConnection.HandleConnectionResult(ConnectTask task, Exception error, Pool pool)
   at XenAdmin.Network.XenConnection.ConnectWorkerThread(Object o)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)

olivierlambert

@dsiminiuk Please use the Markdown syntax for code/text block

```
your code
```

edit: I edited your post but please use it next time.

dsiminiuk

Thanks! will do.

dsiminiuk

I get the Internal error: Xenops_interface.Does_not_exist(_) when I migrate the Windows VM.
I have to restart the toolostack to allow starting of the same VM on the target host in the pool.

borzel

What's about powering off the VM, moving, powering on?

dsiminiuk

@borzel I'll try that. Thanks.

My most recent test was to have all the VMs running on server1, put it into Maint mode and watch all the VMs mmigrate to server2. It behaves the same way with the same error regardless of direction.

dsiminiuk

@borzel If I power off the VM gracefully I can start it on either server in the pool. I have to restart the toolstack on the target server following the error during a live migration and then I can start it on either machine from what appears to be a forcibly power off state (Start windows normally?).

It only happens during a live migration and so far only on this VM which happens to be Windows 2008 R2 and guest tools v7.1.