Import OVA silently fails
-
Hello,
So I exported a VM from VMWare as OVA. I did this test first on a windows 10 install, with no vmware tools installed, exported as OVA and imported into XCP-NG without issue. Then i took a VM that has been running for years, and removed vmware tools, and exported as OVA. When I import it, it shows up importing disk or what have you on the Tasks page, but then after some time it just silently disappears. Nothing shows up in the log.
I have including a screen shot of it importing:
The Selected VM
The Import under-tasks [This runs until a large percent, i think like 30-40% range, i tend to miss exsactly when it fails]
Then when i select the logs tab: Nothing
Is there a physical location i can check for these? /var/logs? or something to see what is causing this import to vaporize?
Any hints would be appreciated.
Thank you,
Andrew -
Ah found something after opening xen center:
They appear to have stopped VMs! Any ideas?
-
More details please --
- Are you running XOA or XO (from sources)?
- What version? Are you up to date?
- Have you tried checking the logs from CLI?
-
- I have tried from XOA (Enterprise) as well from XenCenter.
- Current version: 5.44.1
- What would I look for in these lots, there is alot in here.
Thank you,
-
Alternatively, there's an CLI tool to import an OVA
-
There's a newer version that includes improvements to the OVA import process. You can access it by switching to the
latest
release channel in XOA. -
OK , i updated to lastest, and it has the same effect. Where do i find CLI documentation for importing?
-
-
Ok the file syslog has alot of errors in it, looks like unhandled promise chains. (.then errors) The file is 3MB, how can i get this file to you?
Here is a snippet near the area of import:
Apr 8 01:09:51 xoa xo-server[115382]: 2020-04-08T05:09:51.657Z xo:xapi DEBUG Creating VM Spiceworks-Production Apr 8 01:09:51 xoa xo-server[115382]: 2020-04-08T05:09:51.728Z xo:xapi DEBUG Creating VDI Hard disk 1 on Local storage Apr 8 01:09:51 xoa xo-server[115382]: 2020-04-08T05:09:51.729Z xo:xapi DEBUG Creating VIF for VM Spiceworks-Production on network MGMT-LAN Apr 8 01:09:52 xoa xo-server[115382]: 2020-04-08T05:09:52.538Z xo:xapi DEBUG Creating VBD for VDI Hard disk 1 on VM [Importing...] Spiceworks-Production Apr 8 01:11:07 xoa kernel: [3494938.746302] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=65109 PROTO=2 Apr 8 01:11:07 xoa kernel: [3494938.746397] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=65110 PROTO=2 Apr 8 01:13:17 xoa kernel: [3495068.743182] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=65116 PROTO=2 Apr 8 01:13:17 xoa kernel: [3495068.743286] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=65117 PROTO=2 Apr 8 01:15:27 xoa kernel: [3495198.739905] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=65122 PROTO=2 Apr 8 01:15:27 xoa kernel: [3495198.739971] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=65123 PROTO=2 Apr 8 01:15:54 xoa kernel: [3495225.770190] node invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Apr 8 01:15:54 xoa kernel: [3495225.770191] node cpuset=/ mems_allowed=0 Apr 8 01:15:54 xoa kernel: [3495225.770196] CPU: 1 PID: 115382 Comm: node Not tainted 4.19.0-8-amd64 #1 Debian 4.19.98-1 Apr 8 01:15:54 xoa kernel: [3495225.770196] Hardware name: Xen HVM domU, BIOS 4.11.1-7.8.xcpng8.0 12/12/2019 Apr 8 01:15:54 xoa kernel: [3495225.770197] Call Trace: Apr 8 01:15:54 xoa kernel: [3495225.770206] dump_stack+0x66/0x90 Apr 8 01:15:54 xoa kernel: [3495225.770211] dump_header+0x6b/0x283 Apr 8 01:15:54 xoa kernel: [3495225.770214] ? do_try_to_free_pages+0x2ec/0x370 Apr 8 01:15:54 xoa kernel: [3495225.770215] oom_kill_process.cold.30+0xb/0x1cf Apr 8 01:15:54 xoa kernel: [3495225.770217] ? oom_badness+0xe9/0x140 Apr 8 01:15:54 xoa kernel: [3495225.770219] out_of_memory+0x1a5/0x450 Apr 8 01:15:54 xoa kernel: [3495225.770221] __alloc_pages_slowpath+0xbd8/0xcb0 Apr 8 01:15:54 xoa kernel: [3495225.770223] __alloc_pages_nodemask+0x28b/0x2b0 Apr 8 01:15:54 xoa kernel: [3495225.770225] filemap_fault+0x3bd/0x780 Apr 8 01:15:54 xoa kernel: [3495225.770228] ? alloc_set_pte+0xf2/0x560 Apr 8 01:15:54 xoa kernel: [3495225.770229] ? filemap_map_pages+0x1ed/0x3a0 Apr 8 01:15:54 xoa kernel: [3495225.770282] ext4_filemap_fault+0x2c/0x40 [ext4] Apr 8 01:15:54 xoa kernel: [3495225.770285] __do_fault+0x36/0x130 Apr 8 01:15:54 xoa kernel: [3495225.770286] __handle_mm_fault+0xe6c/0x1270 Apr 8 01:15:54 xoa kernel: [3495225.770288] handle_mm_fault+0xd6/0x200 Apr 8 01:15:54 xoa kernel: [3495225.770291] __do_page_fault+0x249/0x4f0 Apr 8 01:15:54 xoa kernel: [3495225.770294] ? page_fault+0x8/0x30 Apr 8 01:15:54 xoa kernel: [3495225.770295] page_fault+0x1e/0x30 Apr 8 01:15:54 xoa kernel: [3495225.770297] RIP: 0033:0x13a72b9 Apr 8 01:15:54 xoa kernel: [3495225.770303] Code: Bad RIP value. Apr 8 01:15:54 xoa kernel: [3495225.770304] RSP: 002b:00007ffcf8ff0490 EFLAGS: 00010246 Apr 8 01:15:54 xoa kernel: [3495225.770305] RAX: 0000055c072656b9 RBX: 0000000000c0f1f0 RCX: 00000000030be010 Apr 8 01:15:54 xoa kernel: [3495225.770306] RDX: 000000003ab98c20 RSI: 0000000000000007 RDI: 0000000000000001 Apr 8 01:15:54 xoa kernel: [3495225.770307] RBP: 00007ffcf8ff04a0 R08: 000000000000001f R09: 0000000008bf8810 Apr 8 01:15:54 xoa kernel: [3495225.770307] R10: 0000000000000006 R11: 0000000000000127 R12: 00000f8be5127ad9 Apr 8 01:15:54 xoa kernel: [3495225.770308] R13: 00000000030fc400 R14: 0000000000000008 R15: 00007ffcf8ff04e8 Apr 8 01:15:54 xoa kernel: [3495225.770309] Mem-Info: Apr 8 01:15:54 xoa kernel: [3495225.770313] active_anon:330003 inactive_anon:114458 isolated_anon:0 Apr 8 01:15:54 xoa kernel: [3495225.770313] active_file:23 inactive_file:43 isolated_file:0 Apr 8 01:15:54 xoa kernel: [3495225.770313] unevictable:0 dirty:0 writeback:0 unstable:0 Apr 8 01:15:54 xoa kernel: [3495225.770313] slab_reclaimable:8284 slab_unreclaimable:12775 Apr 8 01:15:54 xoa kernel: [3495225.770313] mapped:35 shmem:13 pagetables:2766 bounce:0 Apr 8 01:15:54 xoa kernel: [3495225.770313] free:13146 free_pcp:413 free_cma:0 Apr 8 01:15:54 xoa kernel: [3495225.770315] Node 0 active_anon:1320012kB inactive_anon:457832kB active_file:92kB inactive_file:172kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:140kB dirty:0kB writeback:0kB shmem:52kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Apr 8 01:15:54 xoa kernel: [3495225.770316] Node 0 DMA free:8008kB min:360kB low:448kB high:536kB active_anon:2988kB inactive_anon:4688kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:32kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Apr 8 01:15:54 xoa kernel: [3495225.770318] lowmem_reserve[]: 0 1916 1916 1916 1916 Apr 8 01:15:54 xoa kernel: [3495225.770319] Node 0 DMA32 free:44576kB min:44692kB low:55864kB high:67036kB active_anon:1317024kB inactive_anon:453144kB active_file:376kB inactive_file:524kB unevictable:0kB writepending:0kB present:2072576kB managed:1995344kB mlocked:0kB kernel_stack:4624kB pagetables:11032kB bounce:0kB free_pcp:1652kB local_pcp:300kB free_cma:0kB Apr 8 01:15:54 xoa kernel: [3495225.770322] lowmem_reserve[]: 0 0 0 0 0 Apr 8 01:15:54 xoa kernel: [3495225.770323] Node 0 DMA: 12*4kB (UM) 9*8kB (UM) 5*16kB (UME) 6*32kB (UME) 5*64kB (UE) 5*128kB (UME) 4*256kB (UME) 3*512kB (UE) 0*1024kB 2*2048kB (UM) 0*4096kB = 8008kB Apr 8 01:15:54 xoa kernel: [3495225.770342] Node 0 DMA32: 308*4kB (UMEH) 188*8kB (UMEH) 187*16kB (UMEH) 238*32kB (UMEH) 91*64kB (UEH) 54*128kB (UEH) 27*256kB (UMEH) 4*512kB (UME) 2*1024kB (UM) 4*2048kB (UM) 0*4096kB = 45280kB Apr 8 01:15:54 xoa kernel: [3495225.770351] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Apr 8 01:15:54 xoa kernel: [3495225.770352] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Apr 8 01:15:54 xoa kernel: [3495225.770352] 99 total pagecache pages Apr 8 01:15:54 xoa kernel: [3495225.770353] 31 pages in swap cache Apr 8 01:15:54 xoa kernel: [3495225.770354] Swap cache stats: add 2889276, delete 2889244, find 929577/1183567 Apr 8 01:15:54 xoa kernel: [3495225.770354] Free swap = 0kB Apr 8 01:15:54 xoa kernel: [3495225.770355] Total swap = 2086908kB Apr 8 01:15:54 xoa kernel: [3495225.770356] 522141 pages RAM Apr 8 01:15:54 xoa kernel: [3495225.770356] 0 pages HighMem/MovableOnly Apr 8 01:15:54 xoa kernel: [3495225.770356] 19329 pages reserved Apr 8 01:15:54 xoa kernel: [3495225.770357] 0 pages hwpoisoned
-
This is a RAM issue. Raise your XOA RAM to 4GiB, if it continues, follow: https://xen-orchestra.com/docs/troubleshooting.html#memory
-
After making those changes, it got further, but still halted at about 10%. (I upped to 8GB RAM)
Apr 8 16:38:22 xoa xo-server[467]: 2020-04-08T20:38:22.411Z xo:xapi DEBUG Creating VM Spiceworks-Production Apr 8 16:38:22 xoa xo-server[467]: 2020-04-08T20:38:22.482Z xo:xapi DEBUG Creating VDI Hard disk 1 on Local storage Apr 8 16:38:22 xoa xo-server[467]: 2020-04-08T20:38:22.483Z xo:xapi DEBUG Creating VIF for VM Spiceworks-Production on network MGMT-LAN Apr 8 16:38:23 xoa xo-server[467]: 2020-04-08T20:38:23.294Z xo:xapi DEBUG Creating VBD for VDI Hard disk 1 on VM [Importing...] Spiceworks-Production Apr 8 16:38:27 xoa kernel: [ 66.534931] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=2660 PROTO=2 Apr 8 16:40:37 xoa kernel: [ 196.531337] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=2665 PROTO=2 Apr 8 16:42:47 xoa kernel: [ 326.527746] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:00:18:0a:86:53:c4:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=2671 PROTO=2 Apr 8 16:44:16 xoa xoa-updater[472]: 2020-04-08T20:44:14.045Z xoa-updater WARN onMessage Error: Command failed: xenstore-read vm Apr 8 16:44:16 xoa xoa-updater[472]: fatal error: runtime: out of memory Apr 8 16:44:16 xoa xoa-updater[472]: runtime stack: Apr 8 16:44:16 xoa xoa-updater[472]: runtime.throw(0x8108647, 0x16)
Here is my xo-server.service
ExecStart=/usr/local/bin/node --max-old-space-size=7680 /usr/local/bin/xo-server
Anything else we can try?
-
So there's a leak when you import How big is the OVA?
Can you try with the OVA CLI?
-
The VM is 138GB. I read those instructions, what i am failing to understand is where and how I point it to my local disk? Do you have an example of using a file in C:\Export (on a windows box)?
The example the OVA appears to be local to XOA?
Please advise,
Thank you -
I have tried to use a SR that is used for ISO:
xo-upload-ova --upload Spiceworks-Production.ova 1fee8757-644e-2bce-beb0-9f85380a5842 --override 'nameLabel=test' descriptionLabel='Spiceworks' memory=json:671088640 disks.vmdisk1.descriptionLabel='disk description' disks.vmdisk1.capacity=json:1342177280
The issue of course is, how do i get the settings for disk size and memory. When i try to:
xo-upload-ova --inspect Spiceworks-Production.ova 1fee8757-644e-2bce-beb0-9f85380a5842
It doesn't know where that file is, does it have to be on a Local SR to XOA?
-
Ping @nraynaud who did that
-
I am looking into it on my side. I might come back with questions.
-
OK no problem, i look forward to hearing from you.
-
@nraynaud Any update?
-
I'm still on it, I think I found an unrelated bug whose effect would be to upload a VM without any disk.
-
I think I got the bug, there is some cleanup and QA to be done before releasing a fix.