Veeam and XCP-ng

Andrew

@jbamford Why use Veeam? Because Veeam is an industry standard and certified for use in many GxP businesses. It offers agentless VM backups with existing VMware. It also offers agents for windows backups for standalone workstations (that XO does not). Veeam also supports tapes (LTO). Yes, tapes are still a thing. Like VMware, it's the 900 pound gorilla in the room, so you can't ignore it.

I use XO for backups on XCP (on-site and off-site storage). It's great to have an integrated management and backup solution in one setup.

jbamford

@Andrew Meh if you say so, i mean it is up to you but it will only end in tears.

Making something work for backups which is so crucial is a bad thing.

Tristis Oris

@Andrew veeam is a standart only because other hypervisors have no builtin backups. It not a silver bullet and can't backup everything.

nikade

We're using Veeam for our VMware platform and XOA for our XCP platform and I have to say I prefer XOA. Why may one ask?
Well it is dead simple, its not bloated, you have a ton of options when it comes to configuration, destination, scheduling and so on.

Veeam on the other hand is A LOT faster, we're using a 10G link to our backup site and we're seeing speeds over 7Gbit/s when backing up our VMware platform and I know this is something Vates is working hard on improving in XOA.
Veeam also has application aware backups which is a big deal when you're running MSSQL inside your VM's - I dont think there is any plans from Vates in supporting this and this might be a big deal for customers comming from the VMware and Veeam-side.

jbamford

@nikade that is one of the downfalls. Limit on backup, I have a 10Gbp iSCSI backbone with 10Gbp core network but maximum I am able to achieve is 80mbps maximum turns outs it’s a known problem according to Olivier Citrix Hypervisor also has the same problem.

jasonnix

Hi @jbamford,
So, the Veeam is faster than XO.

jbamford

@jasonnix there is having something faster and there is something that’s going to be reliable. I’d rather compromise on faster backup speed and have the designed backup system.

I’m not saying that Veeam isn’t going to be reliable but what I am going to say is that you are at more risk of backups not being reliable and you are going to have to consult Veeam if you need help resolving a issue, where with XO you are not going to have any problems as it’s designed to work.

Not only that you need to think about extra licensing too.

If this is a Production environment I would go with the XCP-ng and XO because if something goes wrong you are going to be a laughing stock to the company it will be embarrassing. If it’s a homelab then it doesn’t really matter if it breaks.

DustinB

@jasonnix speediness of the backups is certainly a consideration.

As is having an complete ecosystem designed to work together, and not needing to be piecemealed together.

If you want to piecemeal your ecosystem together, that is fine, but you won't find any holistic support from either the XCP-ng community, while you're using Veeam for your backups and attempting to do everything through the CLI.

jasonnix

@DustinB, I like the CLI, because sometimes you don't have access to the graphic environment.

DustinB

@jasonnix said in Veeam and XCP-ng:

@DustinB, I like the CLI, because sometimes you don't have access to the graphic environment.

You are thinking like this why? XO can be deployed in a matter of literal minutes, import your configuration file and you're back in working order.

nikade

@jbamford said in Veeam and XCP-ng:

@nikade that is one of the downfalls. Limit on backup, I have a 10Gbp iSCSI backbone with 10Gbp core network but maximum I am able to achieve is 80mbps maximum turns outs it’s a known problem according to Olivier Citrix Hypervisor also has the same problem.

Yeah it is a known issue that has been around for years, I even reported it on XenServer bugtracker in 2018 and nothing is done about it from Citrix.
I have been dicussing it with @olivierlambert many times and there is some hope that SMAPIv3 will be able to increase the speed.

IMO the speeds are not at "enterprise" level and this is probably a thing that will limit many of the bigger migrations from VMware to XCP-NG and make ppl choose Proxmox instead.

s-master

I wanted to share a personal experience that might be helpful to others, especially if they're transitioning from VMware to XCP-NG like I am. With over 10 years of experience with VMware and about a year with XCP-NG, I found myself facing a challenge due to my lack of familiarity with XCP-NG.

During the transition, I overlooked a crucial detail: the storage provisioning. In VMware, I always used thick provisioning for its perceived speed advantages. However, in XCP-NG, I inadvertently configured one of my hosts with thick provisioning instead of thin provisioning.

This oversight became apparent when the thick-provisioned storage on one of my XCP-NG hosts reached its capacity limit. As a result, I couldn't perform any backups using Xen Orchestra (XO) because there wasn't enough storage available for snapshots.

While I prefer the simplicity of performing backups directly in XO, my mistake necessitated an alternative solution. Fortunately, I had a Veeam license available, and Veeam quickly came to the rescue. Despite my preference for XO, Veeam provided a reliable backup solution in this instance when XO couldn't.

Tristis Oris

@s-master said in Veeam and XCP-ng:

reached its capacity limit

it has nothing to do with xen or provisioning type.

KPS

I do not think, there will be a "Veeam for XCP-ng". I think, we should hope and help to improve XOA.

My (constructive) critic points:

Backup and restore speed should be higher
Instant restore "pendant" is missing --> makes restore speed MORE critical
No real "progressbar" on jobs!
Maintenance-jobs not really visible at all (backup-merge!)

There is also one other thing, but I think, it can be handled mostly with additional agent-based-software:

Application aware backups (active directory, sql, etc.)

--> So the main point is: You are currently not able to estimate the speed or progress of jobs until they are finished and you do not see maintence jobs

s-master

@Tristis-Oris said in Veeam and XCP-ng:

it has nothing to do with xen or provisioning type.

Then what is the root cause of this error below? I mean I can read: 'There is insufficient space.'
As I understand it, if XCP is creating a snapshot on an SR with thick provisioning, it takes up the same amount of storage as the original VDI.
So, I was trapped because of my mistake. But with Veeam and its Linux Agent, I could back it up regardless of insufficient space.
That's what I wanted to convey.

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1709124708488",
  "jobId": "72174171-6b1d-40ff-99f9-847f7c577fd3",
  "jobName": "XXXXX",
  "message": "backup",
  "scheduleId": "95f41044-842f-41c0-85ea-b65089b50f7f",
  "start": 1709124708488,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "e06fd66b-9e4d-8df0-71f4-a8508ab677c1"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "e06fd66b-9e4d-8df0-71f4-a8508ab677c1",
        "name_label": "XXXXX_Debian-10.13"
      },
      "id": "1709124711140",
      "message": "backup VM",
      "start": 1709124711140,
      "status": "failure",
      "tasks": [
        {
          "id": "1709124711152",
          "message": "snapshot",
          "start": 1709124711152,
          "status": "failure",
          "end": 1709124739540,
          "result": {
            "code": "SR_BACKEND_FAILURE_44",
            "params": [
              "",
              "There is insufficient space",
              ""
            ],
            "task": {
              "uuid": "825de1b6-967f-c1dc-1415-c852ef6179c0",
              "name_label": "Async.VM.snapshot",
              "name_description": "",
              "allowed_operations": [],
              "current_operations": {},
              "created": "20240228T12:51:51Z",
              "finished": "20240228T12:52:19Z",
              "status": "failure",
              "resident_on": "OpaqueRef:8f335799-cec0-4024-83de-87a5b44839af",
              "progress": 1,
              "type": "<none/>",
              "result": "",
              "error_info": [
                "SR_BACKEND_FAILURE_44",
                "",
                "There is insufficient space",
                ""
              ],
              "other_config": {},
              "subtask_of": "OpaqueRef:NULL",
              "subtasks": [
                "OpaqueRef:0c1f09de-c43d-4e71-9865-47d7c0dc6b46"
              ],
              "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 80))((process xapi)(filename list.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 122))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 130))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 171))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 209))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 220))((process xapi)(filename list.ml)(line 121))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 222))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 442))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 33))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
            },
            "message": "SR_BACKEND_FAILURE_44(, There is insufficient space, )",
            "name": "XapiError",
            "stack": "XapiError: SR_BACKEND_FAILURE_44(, There is insufficient space, )\n    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_getTaskResult.mjs:11:29)\n    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1006:24)\n    at file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1040:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1030:12)\n    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1203:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1709124739553",
          "message": "clean-vm",
          "start": 1709124739553,
          "status": "success",
          "end": 1709124739561,
          "result": {
            "merge": false
          }
        }
      ],
      "end": 1709124739562,
      "result": {
        "code": "SR_BACKEND_FAILURE_44",
        "params": [
          "",
          "There is insufficient space",
          ""
        ],
        "task": {
          "uuid": "825de1b6-967f-c1dc-1415-c852ef6179c0",
          "name_label": "Async.VM.snapshot",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20240228T12:51:51Z",
          "finished": "20240228T12:52:19Z",
          "status": "failure",
          "resident_on": "OpaqueRef:8f335799-cec0-4024-83de-87a5b44839af",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "SR_BACKEND_FAILURE_44",
            "",
            "There is insufficient space",
            ""
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [
            "OpaqueRef:0c1f09de-c43d-4e71-9865-47d7c0dc6b46"
          ],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 80))((process xapi)(filename list.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 122))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 130))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 171))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 209))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 220))((process xapi)(filename list.ml)(line 121))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 222))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 442))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 33))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
        },
        "message": "SR_BACKEND_FAILURE_44(, There is insufficient space, )",
        "name": "XapiError",
        "stack": "XapiError: SR_BACKEND_FAILURE_44(, There is insufficient space, )\n    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/_getTaskResult.mjs:11:29)\n    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1006:24)\n    at file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1040:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1030:12)\n    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202402270007/packages/xen-api/index.mjs:1203:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    }
  ],
  "end": 1709124739563
}

DustinB

@KPS said in Veeam and XCP-ng:

Application aware backups (active directory, sql, etc.)

So the challenge with this is that currently, there isn't a Windows Signed Driver that is developed by the Vates team for this. You'd be using the driver from xenserver for windows.

This AFAIK hasn't been built for the intention of being backup aware, but for providing an HCL to each guest.

Vates would have to build (either into or their own) guest utility that supports application awareness, which can be done by using the "Driver Development Kit (DDK) 8.0.0"

tc-atwork

@jbamford said in Veeam and XCP-ng:

there is having something faster and there is something that’s going to be reliable

How about fast and reliable, which is Veeam.

I’m not saying that Veeam isn’t going to be reliable but what I am going to say is that you are at more risk of backups not being reliable

If this is a Production environment I would go with the XCP-ng and XO because if something goes wrong

That's a pretty hot take that Veeam, an industry leading product where their only business is backup and they are damn good at it, is going to somehow be less reliable than the, in my opinion, half-baked solution that exists in XO (while I appreciate the work the team is doing to make this better and better, seriously, I love reading the monthly progress updates, the fact remains). Additionally, the fact that XO requires at least double the provisioned storage of a VM in order to properly coalesce snapshots is a drawback that Veeam doesn't require.

While I agree with the answer someone else gave that this question should be getting asked to Veeam, I think the discussion here is important as it may drive interest to Vates to work with Veeam. Especially now that a lot of VMWare customers that are accustomed to all that Veeam can do are making their way to XO, I'd expect Veeam integration to be at least in consideration.

tc-atwork

@DustinB said in Veeam and XCP-ng:

Vates would have to build (either into or their own) guest utility that supports application awareness, which can be done by using the "Driver Development Kit (DDK) 8.0.0"

And they should if they want to compete with VMware and Hyper-V. I think a native, completely Vates-controlled guest utility should be high on the priority list. Preferably, one that is automatically distributed by Windows Update (admittedly, I don't know the cost for this) or automatically installed on Linux systems, controlled by a setting at the VM and/or Pool level.

tc-atwork

@KPS said in Veeam and XCP-ng:

Instant restore "pendant" is missing --> makes restore speed MORE critical

While not entirely the same, depending on the reason you are restoring, you can restore to the most recent snapshot taken by your scheduled backups or setup a separate snapshot schedule independent from your backups.

To add to your list, XO also can't snapshot (and therefore backup) VMs that are configured with USB passthrough due to an XCP-NG 8.2 limitation. Support has told me this "should be solved" in 8.3, but to me should does not mean confirmed and there is no timeframe that I'm aware of for 8.3 stable release.

jbamford

@tc-atwork that’s not entirely true. I’ve had a few clients that has had issues with Veeam where backups get stuck and never finish until you restart Veeam. When having problems like this, this in my opinion is not good enough for backups.

My previous job had 8 Hypervisors in a Cluster which also used Veeam and backups was getting stuck.