VM Creation Failure | "Expected int64, got 'N'\"
-
Good-day Folks,
MY ENVIRONMENT:
- XCP-ng: 3 nodes running v8.3.0
- XOA: v5.98.1 Build 20241024 (air-apped)
- NFS Server: Windows Server 2016
- Networking: Management network of hosts exist within the same VLAN as NFS Server
THE ISSUE:
I'm working through an issue with support, regarding some VDIs disappearing - at a fairly consistent interval - and this has made my lab environment unstable. So, as a remedy, I created a new SR and I'm not having the same issues with VMs created on that SR.So I built a new VM and converted it to a template, then decided to create a new VM from that template. However, I decided not to use the fast copy feature - just to benchmark how long it would take. The VM creation task ran all the way to the end and then failed with the following error:
vm.create { "clone": false, "existingDisks": { "0": { "name_label": "Mail02_Disk1_OS", "name_description": "Operating System Disk for Mail02", "size": 161061273600, "$SR": "921094bf-12a3-78a2-f470-a2b6ad342647" } }, "name_label": "MAIL02", "template": "d7973e98-2211-f806-f6ce-adce6cf4cca8", "VDIs": [ { "name_description": "Data Disk for Mail02", "name_label": "MAIL02_Disk2_Data", "SR": "921094bf-12a3-78a2-f470-a2b6ad342647", "type": "system", "size": 161061273600 } ], "VIFs": [ { "network": "acc073bc-ed2d-adda-c67a-dd68f5b567a9", "allowedIpv4Addresses": [], "allowedIpv6Addresses": [] } ], "CPUs": 4, "cpusMax": 4, "cpuWeight": null, "cpuCap": null, "name_description": "Mail Server #2 (running MailEnable)", "memoryMax": 17179869184, "memoryMin": 8589934592, "memoryStaticMax": null, "autoPoweron": true, "bootAfterCreate": true, "copyHostBiosStrings": false, "createVtpm": false, "destroyCloudConfigVdiAfterBoot": false, "secureBoot": true, "share": false, "coreOs": false, "tags": [ "Member Server" ], "hvmBootFirmware": "uefi" } { "code": "INTERNAL_ERROR", "params": [ "Failure(\"Expected int64, got 'N'\")" ], "call": { "method": "VM.set_memory_static_max", "params": [ "OpaqueRef:5d925d4b-1de3-8ebb-7efa-22c98a4a1cf2", null ] }, "message": "INTERNAL_ERROR(Failure(\"Expected int64, got 'N'\"))", "name": "XapiError", "stack": "XapiError: INTERNAL_ERROR(Failure(\"Expected int64, got 'N'\")) at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12) at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21 at runNextTicks (node:internal/process/task_queues:60:5) at processImmediate (node:internal/timers:447:9) at process.callbackTrampoline (node:internal/async_hooks:128:17)" }The template has a 150GB VDI and I attempted to add a second VDI to the VM at creation, so the process to about 30 minutes to complete, before throwing the error. So thinking that perhaps it may be the copy time that took too long, I removed the second VDI and attempted the creation again. It failed again with the same error.
I welcome any assistance I can get on this, thank you. As it stands now, my Proof of Concept testing has come to a halt, as I'm unable to create any new VMs.
-
kagbasi-ngc said in VM Creation Failure | "Expected int64, got 'N'\":
INTERNAL_ERROR(Failure("Expected int64, got 'N'"))
This is weird, it doesn't ring any bell
I would have a relative confidence on a such unstable setup, the first thing I'd like to rule out is hardware (at least CPU & RAM). Can you run a memtest on this?
-
olivierlambert You mean run a memtest within the OS or via the BIOS? I can certainly do that tonight, I'm in the lab now and will be here for 3-4 hours.
-
olivierlambert So, while troubleshooting this issue, I restarted the Xen Toolstack on the pool master (VMH01).
So I tried creating the VM just to see if perhaps the toolstack restart would've had an effect, and it worked (using fast clone). I then tried without fast clone and it worked as well. Looks like restarting the toolstack on the pool master did the trick - though very concerning why this happened in the first place without any warnings of any underlying issues.
-
Yeah that's weird, but I'm still concerned about the hardware. -
olivierlambert Do you want the memtest on all three hosts, or just the master (VMH01)?
-
olivierlambert So, the memtest of the master host is still ongoing. 1 of 4 passes has completed and no errors have been found. Do you want me to continue and let it run all 4 passes?
One thing that struck me as odd is that even though I have 160GB of RAM installed on this host, when I run
top, it's showing only 2.41GB of total RAM. Is this normal?
-
kagbasi-ngc Yes, top is executed from your Dom0, which is... a VM
When you run memtest, you run it on the entire host. And yes, ideally let it run entirely. -
olivierlambert Awesome, thanks for the explanation - I suspected that. Cool, I'm gonna let it run all 4 passes. Also, should I run the same on the other two hosts?
-
If it's a lab and you can afford the time, at least we'll be on the safe side
-
olivierlambert Cool, I'll run the other two when I get back. I'll kick them off simultaneously.
-
olivierlambert So, as luck would have it, the lab lost partial power around 10pm last night - right when the memtest had reached 98% complete on the 4th pass. Unfortunately, I'm gonna have to run the tests again and let it run over the weekend.
It took about 10 hours to run, so that should be enough hours over the weekend for all three hosts to finish.
-
Ouch
-
olivierlambert yeah, I almost cried when it happened.
-
olivierlambert Memory tests on all three hosts completed successfully; results below.
VMH01 (Master)
VMH02
VMH03
-
Great, so at least we can rule out a memory or CPU issue with memory
-
olivierlambert Yes we can, however, it's unsettling not knowing the root cause. But seems that restarting the Xen Toolstack may have resolved this issue.
I haven't attempted creating a VM yet, so I'll do that when I get in tomorrow.
