@stormi said in XCP-ng 8.0.0 Beta now available!:

You could test with Citrix Hypervisor and then report it at https://bugs.xenserver.org
Maybe they'll want to investigate even if the hardware is not officially supported. If someone can reproduce the issue on supported hardware, that would make things easier.

Though I doubt it was woth the effort, as the support over there is well known (especially for not supported configurations etc.) I did it - for the reason of trying to improve stuff (and giving back to OSS community):
https://bugs.xenserver.org/projects/XSO/issues/XSO-955

@stormi dd stands for disk dump and does exactly that: Copy a stream of data.
Fio however can be configured for precise workloads and read/write mixes, parallel workloads etc.

So the first thing will only give you streamline benchmarks, what almost nobody cares about.
The second can simulate realworld (VM/database...) workloads, where (controller) Caches and non magnetic storage (Flash, Optane, MRAM...) makes the real difference.
Also use big amount of data, since caches can impact small ones extremely. Don't get me wrong: We need them and they can make huge differences, but as long as your benchmarks fully fit into them, it gives your nonsense/fake results. Also (consumer) SSDs start throttling after some 10 to a very few 100 GB of data written. Their caches fill up and they 'overheat'.

You can spend days on benchmarks and how to do what.

https://xen-orchestra.com/blog/xen-orchestra-5-36/

The part about ZFS:

As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create your SR.

Check this documentation for more information about ZFS SR creation.

Once the SR is created, you will have the option to select it during the SR creation directly in Xen Orchestra.

Since it's confusing and technically wrong, it should rather be something like:

As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create a ZFS pool before creating the corresponding SR.

Check this documentation for more information about ZFS SR creation.

Once the ZFS pool is created and mounted, you will have the option to select the mountpoint during the SR creation directly in Xen Orchestra.

...and this forum is missing a "quote" button, as formating text in codeboxes doesn't work (and tags don't survive a linebreak, you have to mark it every line again).

Regards,
Christof

@olivierlambert
Okay, so bascially what I (and other people, I know, who are in a similar environment) need:

• Tape backup is mandatory, as we're talking about (deduplicated, compressed) dozens of TB
• Deduplication and good compression: duplicated, uncompressed ends in PB
• All of that needs to scale! E.g. Commvault scales on 16 threads here even with only 1 - 2 tasks. (Can't say if even more without upgrade of CPU)
  (We need chains, of course, to go B2D2T -> dumping the dedup-store on tape as disaster recovery)
• Applicationawareness (which can be done via agent - agentless is not always king or very important):
  -- MS Exchange, recovering datastores, mailboxes and even single mail items
  -- SQL Servers: MS SQL, MySQL (MariaDB)...
  -- Windows Active Directory Items

Over here I also need to have the option to do backups only inside the VM via agent, as I can't snapshot them.
You might have gotton me wrong: AFAIK quiesced backups always made a VM snapshot and submitted the request also to the VSS writer and then the VM/VHDs have been "read". In this case it will always fail, as there's no space on storage for such. Also Citrix discontinued that in their VM tools! XCP-ng yet lacks a proper tool maintenance with corresponding releases, which would be the only chance to keep that.

As an idea for XOA:

• Make it run on physical hardware, as VMs are too small for such
• You already cooperate with HPE - let them run bare metal on ProLiants!
• ZFS(oL) Could be your way to go to implement dedup and compression

...feel free to add, comment... whatever.

Citrix also has a long history of problematic VM guest-tools. Race conditions with broken drivers, unbootable VMs, broken agents... It's a pain in the a**.
If you like your VMs: Always make a snapshot of every VM before updating the tools.

Is known why they remove the VSS-functionality? I don't get why they remove features (again).

XCP-ng already ate a bunch of time, and I'm sure it'll continue for testing etc
I'm Sysadmin with very limited coding ability - I'm definately out for that.

I guess it would need someone being good at C#.NET. Also XAPI would probably need to know about it (what may already be, as you're implementing it into XOA)...

First rule of all benchmarks: The longer and more often they run, the more precise they are.
If we talk about 1G as base, why should I switch with 10 or 100 to M? That doesn't make any sense at all.

@maxcuttins did you really measure 10 and 100 MB after I said 1 G is not enough for accurate results?

Usually you set blocksize to something usefull, like 1M and set count to e.g. 10000.
Of course you can change blocksize to test a bit, but that's usually between like 64k and maybe 4M.

How about doing your own matrix?

You should consider, that efficiency of vCPUs goes down by each one you add. I don't have the link to that Citrix document handy, so you need to google that.
If you really need that many cores, you should consider a physical machine, which should make a serious bump in performance.
AFAIR it was the overhead of the Xen scheduler, which needs to balance the needs of your VM. The more vCPUs one VM has, the bigger the overhead. I'm sure it didn't change in more recent versions.

Also something to keep in mind: It's not only about NUMA (which is different since 2nd Epyc gen, as they have all memory channels on an IO-Die and only split the caches now), it's also about memory bandwith!

So it adds more complexity and depends on the needs of your workload.
If it benefits from high memory bandwith, a 2nd socket doubles it (technically)!

I've set up a test-environment and indeed it worked. Commvault was (is?) checking for specific version/agent string when connecting to XS/CHV/XCP-ng and in older versions refused to connect to XCP-ng.

I successfully installed the agent on a proxy-vm (it uses a proxy to ro-mount the VM-VHDs and backup the content), connected the pool, delivered the VM-inventory and also ran a successful backup:

In other words: If you're using Commvault (11.28+) you're not any longer locked on Citrix.

@reinvtv said in NFS nconnect support:

As far as I understand it, this will give us multiple tcp streams over an LACP link, truely aggregating traffic on multiple interfaces. (Until now, you needed to use iSCSI multipathing for this, which isnt able to thin provision.).

You're neigher right nor wrong: LACP is more complex than just bundling NICs and in many cases it will NOT give you any benefit.
Why? Simple: For that you need to have NFS using multiple ports but stay with same MAC and IP, which means that your LACP balancing algo needs to be set (and support) L4.
Many just go L2 or L3 (means they decide on MAC or IP-Address).
As a quick search didn't answer me: If it even doesn't go that 'passive range' multiple port way: In typical environments it won't help you at all with LACP.

One might correct my post.

Also interesting: https://vastdata.com/blog/meet-your-need-for-speed-with-nfs/

@florent said in Backup solutions for XCP-ng:

@cg we are envisaging various way, from using iSCSI to access tape from the VM, to using an agent on the tape (but here we'll have to support physical hardware patching , updating ) . There is also a lot of work to ensure we write sequentially without concurrency and to make it work with the futur dedup and to keep a catalog of backups / tapes

I don't know every product, but yet I've never seen a Tapedrive/Library using iSCSI.
iSCSI is usually only used by storage systems, not by devices or libraries.
Common interfaces are either SAS or - especially in larger environments - Fiberchannel. So your way to go, probably, is to passthrough an HBA.

@olivierlambert Sure it is a different thing, that's why I recommended using your connections to HPE to offer a bundle or at least to offer a version, that runs with a (more or less specific) version of one of their servers. As it only makes sense when the environment reaches a certain point, it would make sense to pick a DL380/385 series/generation, which offer a good bandwith of performance and space.
E.g. we use a DL385 with 10x 10 TB HDD + a few SSDs for cache and database.

IMHO it's okay to say: We support bare metal on platform X. Lots of configurations options don't matter for your support, as more memory, bigger CPUs or more storage behind the same controller don't touch the needed drivers/evaluations.

@florent IIRC OpenZFS 2 uses zstd and/or lz4 as efficient algorythms, which do a pretty good job. Yet I only know brotli from webservers.

How do you connect the tape, if it's virtualized?
Putting it on bare metal would also target that (aside of performance benefits and falling restrictions on backup size due to VHD limits).

@olivierlambert
Okay, so bascially what I (and other people, I know, who are in a similar environment) need:

• Tape backup is mandatory, as we're talking about (deduplicated, compressed) dozens of TB
• Deduplication and good compression: duplicated, uncompressed ends in PB
• All of that needs to scale! E.g. Commvault scales on 16 threads here even with only 1 - 2 tasks. (Can't say if even more without upgrade of CPU)
  (We need chains, of course, to go B2D2T -> dumping the dedup-store on tape as disaster recovery)
• Applicationawareness (which can be done via agent - agentless is not always king or very important):
  -- MS Exchange, recovering datastores, mailboxes and even single mail items
  -- SQL Servers: MS SQL, MySQL (MariaDB)...
  -- Windows Active Directory Items

Over here I also need to have the option to do backups only inside the VM via agent, as I can't snapshot them.
You might have gotton me wrong: AFAIK quiesced backups always made a VM snapshot and submitted the request also to the VSS writer and then the VM/VHDs have been "read". In this case it will always fail, as there's no space on storage for such. Also Citrix discontinued that in their VM tools! XCP-ng yet lacks a proper tool maintenance with corresponding releases, which would be the only chance to keep that.

As an idea for XOA:

• Make it run on physical hardware, as VMs are too small for such
• You already cooperate with HPE - let them run bare metal on ProLiants!
• ZFS(oL) Could be your way to go to implement dedup and compression

...feel free to add, comment... whatever.

@olivierlambert said in Backup solutions for XCP-ng:

@cg This forum is not focused on XOA sales, it's a community forum where people (at Vates but not exclusively) help others for free when they can on any XO version (source or paid). Same for XCP-ng. And my proof is all other topics this years here.

Our official doc is also listing alternative backup solutions. So no, don't try to imply this too, this is simply false

It's more a "partner page". It's missing a feature matrix, something like you did with VMware. Also there are more solutions like Unitrends, Alike and SEP - probably a few more, I don't know of. SEP also supports dedup and is application aware of (MS) SQL and Exchange - at least at the point I was evaluating a new solution. Though I had a massive performance issue and didn't get support to find the reason.

You can't just pop in there and tell there's no powerful solution to backup XCP-ng in general, which is dishonest. If it doesn't fit your use case it's something very different, and we are always happy to spend some time to learn about what feature is missing, that's how we built XOA and XCP-ng from the start.

I would say: Of course I can and I'm right, but I'm fine if we agree on: We have different levels of expectations about the features a solutions needs.
If something isn't very powerful, it doesn't mean it's bad or doesn't fit for anyone.
My usecase is a common one, but there are many you don't hear from, because they don't even seriously consider moving away from VMware, which is pretty much the standard in my type of environment.

I'll move to PM as this pretty much didn't go to a "discuss backup solutons for XCP-ng", instead a very specific thread about my (and collegues) expectations and your POV.

But we might change to private conversion about the details, if you're interested.
Generally: This is the XCP-ng forum and should not be focussed on XOA sales, so IMHO it would be good for XCP-ng to give an overview with a feature matrix or something like that.

There are enough people that just don't even give solutions like this a chance, when they can't (easily) see their options.
And by that you might miss XCP-ng pro customers. If someone pays for (enterprise) backup software, he most likely won't run the hypervisors unsupported.

@olivierlambert said in Backup solutions for XCP-ng:

Quoting you:

anyone is considering XCP-ng for his enterprise environment, but was missing a powerfull backup solution

This implies clearly there's no existing powerful backup solution for XCP-ng.

Okay, you somehow got me there: I implied someone is using XCP-ng to host a bigger Windows environment, which usually includes MS SQL DBs, Exchange and, of course, an Active Directory. And for that, the application awareness is mandatory at some point.