vm import hangs, and does not complete

Jonathon

Hello!

I have done some searching for this issue, if I have missed it I am very sorry!

I am attempting to have an offline, encrypted backup of a vm. This vm will only be brought online in the rare event it is needed. I have created an xva export, however when I attempt to import nothing happens.

It takes a couple minues, and then I see the upload traffic on my end, but nothing happens on xoa and then it times out after 10 minutes.

This is the only error I see in xo-server. I do not see anything in tasks or logs in xoa.

Sep 12 16:55:51 xoa xo-server[1009]: 2023-09-12T16:55:51.839Z xo:xo ERROR HTTP request error {
Sep 12 16:55:51 xoa xo-server[1009]:   data: { filter: undefined, limit: undefined },
Sep 12 16:55:51 xoa xo-server[1009]:   error: NodeError: Premature close
Sep 12 16:55:51 xoa xo-server[1009]:       at ServerResponse.onclose (/opt/xen-orchestra/packages/xo-server/node_modules/readable-stream/lib/internal/streams/end-of-stream.js:121:76)
Sep 12 16:55:51 xoa xo-server[1009]:       at ServerResponse.emit (node:events:526:35)
Sep 12 16:55:51 xoa xo-server[1009]:       at ServerResponse.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Sep 12 16:55:51 xoa xo-server[1009]:       at emitCloseNT (node:_http_server:996:10)
Sep 12 16:55:51 xoa xo-server[1009]:       at Socket.onServerResponseClose (node:_http_server:278:5)
Sep 12 16:55:51 xoa xo-server[1009]:       at Socket.emit (node:events:526:35)
Sep 12 16:55:51 xoa xo-server[1009]:       at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Sep 12 16:55:51 xoa xo-server[1009]:       at TCP.<anonymous> (node:net:323:12)
Sep 12 16:55:51 xoa xo-server[1009]:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17),
Sep 12 16:55:51 xoa xo-server[1009]:   fn: 'handleGetAllObjects'
Sep 12 16:55:51 xoa xo-server[1009]: }
lines 2056458-2056529/2056529 (END)

Danp

Hi... It appears that you are using XO built from the sources. When did you last update?

Jonathon

@Danp

Recently

olivierlambert

You need to get the latest commit on master in any case, even if you are "only" 6 commits behind, it's still needed to be on HEAD to help you

Jonathon

@olivierlambert Done, no change in behaviour. Still nothing in Tasks or Logs pages.

olivierlambert

What's your node version?

Jonathon

@olivierlambert

$ node --version
v18.17.1

Jonathon

OMG, it is somehow Cloudflare zero trust access.

We have our XOA behind ZT, so on a whim I tried importing the vm export directly in our internal network, and it works.

Sorry for the goose chase, maybe this will help someone in the future. May update this when we have some better idea of what the problem actually is.

olivierlambert

So Cloudflare cut the HTTP connection? Oo

john.c

@olivierlambert @Jonathon It must be that your system's connection to the instance (from Internet over Cloudflare's Zero Trust service) violates one of the principals of Zero Trust (ZT). It likely be the workloads part of "Automate Context Collection And Response".

Especially given that you were accessing XOA and importing a VM. If the VM being imported is too great a risk, under ZT then disconnect (cut the connection), to stop the risky operation from proceeding and spreading across the rest of the network.

Especially if the VM is an unknown quantity, such as a bespoke VM specific to your business, or home lab.

It may be worth bringing Cloudflare into this conversation, as given the size of Vates so that XOA and XCP-ng can be prepped for ZT. As likely operations for VM Importing violates a principle of ZT and/or Cloudflare's implementation of such.

There maybe other places where XOA violates ZT principals, so maybe worth having the code audited and validated for vulnerabilities and certified for ZT readiness.

https://www.cloudflare.com/en-gb/zero-trust/
https://www.crowdstrike.com/cybersecurity-101/zero-trust-security/

olivierlambert

I'm pretty much certain that CloudFlare has no idea whatsoever about what's happening (VM import).

john.c

@olivierlambert No they most likely would have an idea, due to the cut off! As the workloads and principles and implementation of Zero Trust requires a real-time visibility into 100s of user and application identity attributes, as well as workloads.

Then to implement the appropriate response (cut the connection or another response) if the risk is too high, requires enough data and also records to maintain compliance with regulatory security standards.

@Jonathon Checkout and gather the logs on the Cloudflare account dashboard for that instance of the ZT service. It may contain further information about the cut connection, if you can't interpret it then Cloudflare support may be able to help.

olivierlambert

Do you have any proof on that? VM import is XO is very specific and still a niche product, I'll be surprised that CF has any idea on what's going on except a surge of traffic.

john.c

@olivierlambert I have asked Jonathon to gather logs for his instance of that Cloudflare ZT service. As it will highly likely contain the evidence, due to the cut off event.

Cause if XOA can be used without doing that action, then it was that action which caused this via CloudFlare ZT.

Then there's the situation that XOA is a niche product (your words), as thus in this case it may be relatively unknown. So something unknown is likely considered highly risky, especially if the operation (process) is highly risky. Importing a new VM and if it is of something unknown then it's also highly risky.

This is re-enforced by the fact that they were able to import the VM internally without going over Cloudflare's ZT service.

So if the risk factor reaches a certain limit (specific to the service and account) then cut off (drop) the connection, to secure the network.

Jonathon

I do not see any logs or admin panel that insinuates that it is actively being blocked.

john.c

@Jonathon @olivierlambert If it is not being blocked, then it may be their efforts to prevent node saturation by moving from node to node. What plan are you on, is it the free one?

If so then this movement will occur more frequently as they move from node to node roughly around every 5-10 minutes. So will experience this loss (timeout) of connection on a regular basis! With a paid plan its much less frequently, also the Cloudflare ZT support team with detailed logs may be able to aid in stabilising connectivity.

https://community.cloudflare.com/t/my-remote-users-drop-at-least-5-times-a-day-randomly/474817
https://community.cloudflare.com/t/cloudflare-ssh-proxy-dropping-connections/397306

Their minimum paid plan with 100% uptime and SLA is the "Pay-as-you-go" plan. So if your going to be carrying out actions in XOA regularly over a remote connection, which need to have a long timeout then a paid plan would pay for itself.