Veeam and XCP-ng
-
@jbamford said in Veeam and XCP-ng:
@nikade that is one of the downfalls. Limit on backup, I have a 10Gbp iSCSI backbone with 10Gbp core network but maximum I am able to achieve is 80mbps maximum turns outs it’s a known problem according to Olivier Citrix Hypervisor also has the same problem.
Yeah it is a known issue that has been around for years, I even reported it on XenServer bugtracker in 2018 and nothing is done about it from Citrix.
I have been dicussing it with @olivierlambert many times and there is some hope that SMAPIv3 will be able to increase the speed.IMO the speeds are not at "enterprise" level and this is probably a thing that will limit many of the bigger migrations from VMware to XCP-NG and make ppl choose Proxmox instead.
-
I wanted to share a personal experience that might be helpful to others, especially if they're transitioning from VMware to XCP-NG like I am. With over 10 years of experience with VMware and about a year with XCP-NG, I found myself facing a challenge due to my lack of familiarity with XCP-NG.
During the transition, I overlooked a crucial detail: the storage provisioning. In VMware, I always used thick provisioning for its perceived speed advantages. However, in XCP-NG, I inadvertently configured one of my hosts with thick provisioning instead of thin provisioning.
This oversight became apparent when the thick-provisioned storage on one of my XCP-NG hosts reached its capacity limit. As a result, I couldn't perform any backups using Xen Orchestra (XO) because there wasn't enough storage available for snapshots.
While I prefer the simplicity of performing backups directly in XO, my mistake necessitated an alternative solution. Fortunately, I had a Veeam license available, and Veeam quickly came to the rescue. Despite my preference for XO, Veeam provided a reliable backup solution in this instance when XO couldn't.
-
@s-master said in Veeam and XCP-ng:
reached its capacity limit
it has nothing to do with xen or provisioning type.
-
I do not think, there will be a "Veeam for XCP-ng". I think, we should hope and help to improve XOA.
My (constructive) critic points:
- Backup and restore speed should be higher
- Instant restore "pendant" is missing --> makes restore speed MORE critical
- No real "progressbar" on jobs!
- Maintenance-jobs not really visible at all (backup-merge!)
There is also one other thing, but I think, it can be handled mostly with additional agent-based-software:
- Application aware backups (active directory, sql, etc.)
--> So the main point is: You are currently not able to estimate the speed or progress of jobs until they are finished and you do not see maintence jobs
-
@Tristis-Oris said in Veeam and XCP-ng:
it has nothing to do with xen or provisioning type.
Then what is the root cause of this error below? I mean I can read: 'There is insufficient space.'
As I understand it, if XCP is creating a snapshot on an SR with thick provisioning, it takes up the same amount of storage as the original VDI.
So, I was trapped because of my mistake. But with Veeam and its Linux Agent, I could back it up regardless of insufficient space.
That's what I wanted to convey.{ "data": { "mode": "full", "reportWhen": "failure" }, "id": "1709124708488", "jobId": "72174171-6b1d-40ff-99f9-847f7c577fd3", "jobName": "XXXXX", "message": "backup", "scheduleId": "95f41044-842f-41c0-85ea-b65089b50f7f", "start": 1709124708488, "status": "failure", "infos": [ { "data": { "vms": [ "e06fd66b-9e4d-8df0-71f4-a8508ab677c1" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "e06fd66b-9e4d-8df0-71f4-a8508ab677c1", "name_label": "XXXXX_Debian-10.13" }, "id": "1709124711140", "message": "backup VM", "start": 1709124711140, "status": "failure", "tasks": [ { "id": "1709124711152", "message": "snapshot", "start": 1709124711152, "status": "failure", "end": 1709124739540, "result": { "code": "SR_BACKEND_FAILURE_44", "params": [ "", "There is insufficient space", "" ], "task": { "uuid": "825de1b6-967f-c1dc-1415-c852ef6179c0", "name_label": "Async.VM.snapshot", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20240228T12:51:51Z", "finished": "20240228T12:52:19Z", "status": "failure", "resident_on": "OpaqueRef:8f335799-cec0-4024-83de-87a5b44839af", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "SR_BACKEND_FAILURE_44", "", "There is insufficient space", "" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [ "OpaqueRef:0c1f09de-c43d-4e71-9865-47d7c0dc6b46" ], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 80))((process xapi)(filename list.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 122))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 130))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 171))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 209))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 220))((process xapi)(filename list.ml)(line 121))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 222))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 442))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 33))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "SR_BACKEND_FAILURE_44(, There is insufficient space, )", "name": "XapiError", "stack": "XapiError: SR_BACKEND_FAILURE_44(, There is insufficient space, )\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_getTaskResult.mjs:11:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1006:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1040:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1030:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1203:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1709124739553", "message": "clean-vm", "start": 1709124739553, "status": "success", "end": 1709124739561, "result": { "merge": false } } ], "end": 1709124739562, "result": { "code": "SR_BACKEND_FAILURE_44", "params": [ "", "There is insufficient space", "" ], "task": { "uuid": "825de1b6-967f-c1dc-1415-c852ef6179c0", "name_label": "Async.VM.snapshot", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20240228T12:51:51Z", "finished": "20240228T12:52:19Z", "status": "failure", "resident_on": "OpaqueRef:8f335799-cec0-4024-83de-87a5b44839af", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "SR_BACKEND_FAILURE_44", "", "There is insufficient space", "" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [ "OpaqueRef:0c1f09de-c43d-4e71-9865-47d7c0dc6b46" ], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 80))((process xapi)(filename list.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 122))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 130))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 171))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 209))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 220))((process xapi)(filename list.ml)(line 121))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 222))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 442))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 33))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "SR_BACKEND_FAILURE_44(, There is insufficient space, )", "name": "XapiError", "stack": "XapiError: SR_BACKEND_FAILURE_44(, There is insufficient space, )\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_getTaskResult.mjs:11:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1006:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1040:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1030:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1203:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1709124739563 } -
@KPS said in Veeam and XCP-ng:
Application aware backups (active directory, sql, etc.)
So the challenge with this is that currently, there isn't a Windows Signed Driver that is developed by the Vates team for this. You'd be using the driver from xenserver for windows.
This AFAIK hasn't been built for the intention of being backup aware, but for providing an HCL to each guest.
Vates would have to build (either into or their own) guest utility that supports application awareness, which can be done by using the "Driver Development Kit (DDK) 8.0.0"
-
@jbamford said in Veeam and XCP-ng:
there is having something faster and there is something that’s going to be reliable
How about fast and reliable, which is Veeam.
I’m not saying that Veeam isn’t going to be reliable but what I am going to say is that you are at more risk of backups not being reliable
If this is a Production environment I would go with the XCP-ng and XO because if something goes wrong
That's a pretty hot take that Veeam, an industry leading product where their only business is backup and they are damn good at it, is going to somehow be less reliable than the, in my opinion, half-baked solution that exists in XO (while I appreciate the work the team is doing to make this better and better, seriously, I love reading the monthly progress updates, the fact remains). Additionally, the fact that XO requires at least double the provisioned storage of a VM in order to properly coalesce snapshots is a drawback that Veeam doesn't require.
While I agree with the answer someone else gave that this question should be getting asked to Veeam, I think the discussion here is important as it may drive interest to Vates to work with Veeam. Especially now that a lot of VMWare customers that are accustomed to all that Veeam can do are making their way to XO, I'd expect Veeam integration to be at least in consideration.
-
@DustinB said in Veeam and XCP-ng:
Vates would have to build (either into or their own) guest utility that supports application awareness, which can be done by using the "Driver Development Kit (DDK) 8.0.0"
And they should if they want to compete with VMware and Hyper-V. I think a native, completely Vates-controlled guest utility should be high on the priority list. Preferably, one that is automatically distributed by Windows Update (admittedly, I don't know the cost for this) or automatically installed on Linux systems, controlled by a setting at the VM and/or Pool level.
-
@KPS said in Veeam and XCP-ng:
- Instant restore "pendant" is missing --> makes restore speed MORE critical
While not entirely the same, depending on the reason you are restoring, you can restore to the most recent snapshot taken by your scheduled backups or setup a separate snapshot schedule independent from your backups.
To add to your list, XO also can't snapshot (and therefore backup) VMs that are configured with USB passthrough due to an XCP-NG 8.2 limitation. Support has told me this "should be solved" in 8.3, but to me should does not mean confirmed and there is no timeframe that I'm aware of for 8.3 stable release.
-
@tc-atwork that’s not entirely true. I’ve had a few clients that has had issues with Veeam where backups get stuck and never finish until you restart Veeam. When having problems like this, this in my opinion is not good enough for backups.
My previous job had 8 Hypervisors in a Cluster which also used Veeam and backups was getting stuck.
-
@tc-atwork said in Veeam and XCP-ng:
@DustinB said in Veeam and XCP-ng:
Vates would have to build (either into or their own) guest utility that supports application awareness, which can be done by using the "Driver Development Kit (DDK) 8.0.0"
And they should if they want to compete with VMware and Hyper-V. I think a native, completely Vates-controlled guest utility should be high on the priority list. Preferably, one that is automatically distributed by Windows Update (admittedly, I don't know the cost for this) or automatically installed on Linux systems, controlled by a setting at the VM and/or Pool level.
Right, I don't disagree that this should be a very high priority, but as the saying goes "one does not make many". (or something like that).
Using the XenServer driver is likely quite sufficient for many others, and while that doesn't address the needs of everyone it's likely covering the bulk of the paying customers today, which those customers get to vote on different features.
-
J jasonnix marked this topic as a question on
-
J jasonnix has marked this topic as solved on
-
@jasonnix XOA does the primary things Veeam does. Backups, backup copies, full/incremental replication, and file level restores. The only thing I've ran into that XOA can't do is something like restoring active directory objects directly to a domain controller, or native SQL Server level restores without having to restore the whole VM. Those are the only instances where I'd consider running the Veeam agent directly. XOA works perfectly otherwise.
