1. Home
  2. Xen Orchestra

Subcategories

  • VMs, hosts, pools, networks and all other usual management tasks.

    399 Topics
    3k Posts
    M
    @Danp @DustinB @ph7 This host does not run any VMs, just used for CR I've increased the dom0 ram to 4GB with no more alarms. [image: 1760735454438-14d145a4-ed5f-40c9-9d97-f9fa5da99023-image.png]

  • ACLs, Self-service, Cloud-init, Load balancing...

    95 Topics
    805 Posts
    olivierlambertO
    Adding @pdonias in the loop

  • All XO backup features: full and incremental, replication, mirrors...

    410 Topics
    4k Posts
    T
    Sorry in advance for the long-winded post but I am experiencing some long backup times and trying to understand where the bottleneck lies and if there’s anything that I could change to improve the situation. We’re currently using backups through the XOA, backing up to remotes on a Dell Data Domain via NFS. The backup is configured to use an XOA backup proxy for this job to keep the load off of our main XOA. As an example we have a delta backup job configured for a pool and it backs up about 100 VMs. We have our concurrency set to 16, we use NBD and changed block tracking and we merge backups synchronously. The last backup for this job took 15 hours and moved just over 2 TiB of data. After examining the logs from this backup (downloading the json and converting to excel format for easier analysis) I found that there are 4 distinct phases for each VM backup: an initial clean, a snapshot, a transfer and a final clean. I have also found that the final clean phase takes by far the most amount of time on each backup. The Initial Clean Duration time for each server was typically somewhere between a couple seconds and 30 seconds. The Snapshot Duration was somewhere between 2-10 minutes per VM. The Transfer Duration varied between a few seconds and around 30 minutes. The Final Clean Duration however was anywhere between 25 minutes on the low end to almost 5 hours on the high end. The amount of time that this phase took was not proportional to the disk size of the vm being backed up or the transfer size for the backup. I found 2 VMs, each with a single 100GB hard disk and both moved around 20GB of changed data. One of them experienced a Final Clean Duration of 30 minutes and the other was 4 hours and 30 minutes in the same backup job. We also have a large vmware infrastructure and use Dell Power Protect to backup the VMs there to the same Data Domain and we do not see similar issues with backup times in that system. So that got me thinking what the differences were between them and how some of those differences might be affecting the backup job duration. One of the biggest differences that I could come up with was the fact that Power Protect uses the DDBoost protocol to communicate with the Data Domains whereas we had to create NFS exports from the Data Domain to use as backup remotes in Xen Orchestra. Since DDBoost uses client side deduplication it significantly cuts down on the amount of data transferred to the Data Domain. But our transfer time wasn’t the bottleneck here, it was the final clean duration time. This led me to investigate what is actually happening during this phase and please correct me if I’m wrong but it seems like when XO performs coalescing over NFS after the backup: The coalescing process reads each modified block from the child VHD and writes it back to the parent VHD. Over NFS, this means: Read request travels to Data Domain Data Domain reconstructs the deduplicated block (rehydration) Full block data travels back to the proxy (or all the way back to the the xcp-ng host, I’m not entirely sure on this one) xcp-ng processes the block Full block data travels back to Data Domain Data Domain deduplicates it again (often finding it's duplicate) So it seems that the Data Domain must constantly rehydrate (reconstruct) deduplicated data for reads, only to immediately deduplicate the same data again on writes. With DDBoost, it seems like this cycle doesn't happen because the client already knows what's unique. So it seems that each write during coalescing potentially triggers: Deduplication processing Compression operations Copy-on-write overhead for already deduplicated blocks This happens for every block during coalescing, even though most blocks haven't actually changed. So I guess I have a few questions. Is anyone else using NFS to a Data Domain as a backup target for backups in Xen Orchestra and if so have you seen the same kind of performance? For others that backup to a target device that doesn’t handle inline dedup and compression do you see the same or better performance from your backup job times? Does Vates have any plans to incorporate the DDBoost library as an option for the supported protocols when connecting a backup remote? Is there any expectation that the qcow2 disk format could help with this at all vs vhd format?

  • Everything related to Xen Orchestra's REST API

    69 Topics
    544 Posts
    olivierlambertO
    I will let @mathieuRA to answer

  • Terraform, Packer or any tool to do IaC

    38 Topics
    367 Posts
    sidS
    @HolgiB said in Better / more flexible way to add and edit CloudInit templates in XO ?: All this Terraform / Open Tofu stuff is nice but I guess generating VMs via Cloud Init and XO will be the entry level for everyone before trying out a much bigger infrastructure as code solution, right ? Starting to use Terraform is not a big step, and it's how I manage even a small setup with < 20 VMs, though not everything can be managed that way, for example SRs cannot currently be created through terraform, so XO is still needed too. those init files often are technically correct but still fail for some strange reason I agree, debugging cloud-init is not a fun task. I don't know if it is an approach you could take, but in my case I keep my cloud-init extremely simple. It only sets up networking and a single fixed user account with an SSH key. Then other tooling, for example Ansible, takes over from there to configure the VM.
Log in to post
Load new posts
  • C

    NFS Storage Leads to File System Errors After Unplanned Shutdown

    6
    0 Votes
    6 Posts
    620 Views
    I
    @creoleMalady Fingers crossed And don't feel stupid. Everybody has those moments - even after decades of experience.
  • A

    Disk import failed

    Solved
    10
    1
    0 Votes
    10 Posts
    729 Views
    olivierlambertO
    Great news! Thanks for the feedback.
  • D

    XOA Console not coming up

    6
    0 Votes
    6 Posts
    553 Views
    D
    @olivierlambert Tried this - XCPNG | ~/_scripts > netstat -tulpn | sed -n '1,2p;/5900\|80/p' Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:5900 0.0.0.0:* LISTEN 1741/vncterm tcp 0 0 0.0.0.0:10809 0.0.0.0:* LISTEN 2896/xapi-nbd tcp6 0 0 :::80 :::* LISTEN 2887/xapi udp 0 0 0.0.0.0:780 0.0.0.0:* 1029/rpcbind udp6 0 0 :::780 :::* 1029/rpcbind VNC on the XCPNg box is listening to 127.0.01 : 5900, may be - I should change this to listen to 0.0.0.0 ? Also, something to do with protocol used - the browser > inspect > network - seems to use ws:// protocol and receives a 101 - upgrade, should it have been vnc ? ( apologies if its a random guess rant here) -- Attached screenshot of browser console logs. FYI - 192.168.0.49:3000 - is where my XOA VM is running as a VM on XCP Ng (192.168.0.45) and the logs below is when the console for itself (i.e. same IP) [image: 1751716444788-selection_157.png]
  • A

    XOA vs XO vs Backup feature

    4
    0 Votes
    4 Posts
    442 Views
    A
    @kagbasi-ngc Hi, thanks for your thoughts. I get on well with Linux myself - I’m using XO from source following the documentation, and actually, Ronivay’s script, as you mentioned, makes it all even handier. Still, I can’t help but think - your average home user, a total amateur, is just going to land on the XCP-ng host homepage and click "Deploy XOA". And then they’ve no backup, outside of the trial period. But sure, if XOA is aimed squarely at business and enterprise users with paid licences, fair enough that makes perfect sense. I just feel like backup isn’t really a purely business or enterprise feature, unlike, say, proxy instances or hyper-converged storage. It's something even home users would genuinely benefit from. But as it is mentioned above, that’s just how it’s set up - and like you said yourself, everyone’s got the chance to learn something new. And sure, in the age of AI chatbots, there’s really no excuse not to manage it
  • B

    XO and XCP-ng pricing

    34
    0 Votes
    34 Posts
    5k Views
    B
    @DustinB Been using it since 2012, non-profits fell under the academic tier and it was originally by number of cpu which was 2 cpu per host, under 192G ram but again didnt include vmotion etc, just plain ole esxi why I used Veeam since it brought replication into the mix again still using them at a very low cost. They had essentials and essentials plus and the academic pricing was very low, my understanding is they would rather you use VMware vs something else. Again I have receipts going back to 2012, pretty much same price for all these years until Broadcom entered the picture. There was nothing sus about it, you had to verify your non-profit category and fit within the limts, it was pretty standard pricing for a low number of cores, I could go to CDW or any reseller and get the same pricing. Why you have so many small companies jumping ship many were using vmware essentials which was WAY cheaper then what they are doing now which the only option is Standard and it now requires min of 16 cores per host @$50/core. That is the new and wonderful world of Broadcom... Anyway, this was pretty common stuff under VMware, why so many folks were using it, even the small shops.
  • K

    Some weird Alt Gr issue

    3
    0 Votes
    3 Posts
    200 Views
    K
    @olivierlambert thank you for your response. Just tested with another laptop : no more issue... Well, we will inform our future users and wait for v6 !
  • P

    Transport-email plugin - oauth for gmail

    6
    0 Votes
    6 Posts
    658 Views
    D
    @Pierre-Briec You can try something like Mailjet.
  • bogikornelB

    XO SocketError: other side closed

    Solved
    12
    0 Votes
    12 Posts
    711 Views
    olivierlambertO
    Excellent news! Thanks for keeping us posted
  • D

    "Management agent not detected" and "No Xen tools detected"

    guest tools windows windows server drivers
    14
    1
    0 Votes
    14 Posts
    4k Views
    TheNorthernLightT
    @dinhngtu Thank you very much for this. Luckily, these last few 2012 OS's will be mothballed at the end of August, but... sometimes schedules and security don't align.
  • R

    XO one time job scheduler

    3
    0 Votes
    3 Posts
    266 Views
    tjkreidlT
    @RS One option would be this, assuming in this se you want to run the job at midnight on Dec. 25: /bin/echo "/path/to/your/script.sh" | at midnight Dec 25 While cron doesn't offer a specific one-time execution, you could also do this in cron but would have to remove the entry afterwards: 0 0 25 12 * /path/to/your/script.sh Also, take a look at this option: https://www.fastcron.com/guides/one-time-cronjobs/
  • D

    CPU Stats bottoming out to Zero every five minutes

    10
    1
    0 Votes
    10 Posts
    710 Views
    tjkreidlT
    @DKirk That all makes sense, thanks for clarifying. Looks like there are further comments below that seem to pinpoint where the issue may lay. The key point you make is only "after the last updates" is when this started happening!
  • A

    VGA on Xen Orchestra

    12
    1
    0 Votes
    12 Posts
    1k Views
    A
    Olivier, I can say it's worse without the VGA being selected to 16Mb. That option is pretty much a must for any desktop virtualization. The novelty for me is people get far above 1080...
  • P

    Feedback XO v6

    7
    6
    0 Votes
    7 Posts
    769 Views
    P
    @lsouai-vates Today I noticed that there is the same in xo5 (commit 25192) Jobs with disabled jobs but scheduled in a sequence job are shown with Not backed up VMs [image: 1747997034588-9d25866e-a8b8-4bd6-85ac-bcb5973fd6fc-image.png] By the BCK_Running tag, it is backed up [image: 1747997147334-f5471bb0-7600-4d6e-ad81-da9c13dfa6bd-image.png]
  • C

    Display Slowness/Lag

    5
    0 Votes
    5 Posts
    508 Views
    C
    @ph7 I'm using local software mirror with mdadm. Its both boot and vm storage.
  • B

    Connection failed - Unknown error

    Solved
    6
    1
    0 Votes
    6 Posts
    703 Views
    G
    @olivierlambert Thank you for your response. I have followed most of the documentation on setting up HA, and made sure that I have 3 hosts. All are Dell 7090 64GB RAM with 2 dedicated HA NICS. XCP-ng 8.3 is patched, XO is up to date, Xen tools 9.4.0 on all Windows VMs, and using the Xen tools ISO that can be mounted. eth0 1Gbs management network eth1 10Gbs iSCSI network DAC eth2 10Gbs VLAN Trunk, including backup network eth3 1Gbs HA eth4 1Gbs HA Overall, XCP-ng and XO are doing an excellent job. I feel like I am on an Easter egg hunt when pouring through documentation, but that is mostly my problem, because I see that you have put a lot of effort into organizing and updating it. My "use case" is replacing the VMware ESXi/vSphere 2-node cluster with XCP-ng and XO at the small high school (non-profit) where I volunteer as the IT guy. I feel that I am at the point where I trust XCP-ng and XO enough to make the switch during the summer break. Sorry to run on. The yellow triangles behind the non-master hostnames are just a small nuisance. I do notice that the hosts are hard to shutdown. Sending a shutdown command from either XO or from the host console causes a host to reboot, but not shutdown, even after having previously entered maintenance mode. I feel that this is not a big issue, just another small nuisance. The XO cluster is maintaining the integrity of the guest VMs. I will be reading all of the documentation I can, and upgrading to 8.4 when it is available. Thank you for supporting your awesome hypervisor.
  • ForzaF

    Retention for log files and audit logs

    5
    0 Votes
    5 Posts
    452 Views
    ForzaF
    @Bastien-Nollet Thanks for checking. At least I know it is set to 20000 entries at the moment. Thank you.
  • A

    RAID Status on the Advanced tab of the host

    10
    1
    0 Votes
    10 Posts
    849 Views
    olivierlambertO
    The plugin was initially created to check RAID created during install, not custom RAIDs created manually. But that could be an improvement to the plugin @stormi I leave you the honor to add this somewhere
  • V

    Cannot get to XO Lite portal

    7
    0 Votes
    7 Posts
    822 Views
    olivierlambertO
    @Virtual said in Cannot get to XO Lite portal: @olivierlambert So my host IP is 192.168.1.40 and the Xen orchestra IP is 192.168.1.45 To clarify, I was referring to XO 6 Lite. Going to https://192.168.1.40 brings up the page with XoA quick deploy, cli documentation, etc. It doesn’t navigate to XO 6 lite Going to https://192.168.1.45/v6 still gives me the error mentioned in the original post Indeed, as said, there's no such thing as "XO Lite 6". XO Lite is simply "XO Lite" (we'll have a v1 at some point, v6 is like far far in the future ) So yes, https://192.168.1.40 should display XO Lite if you use XCP-ng 8.3 (you can install it manually for 8.2 but it's not bundled by default) And https://192.168.1.45/v6 should display XO 6, only if you built it correctly to get XO 6 components built.
  • B

    cant create a private cross pool network

    3
    0 Votes
    3 Posts
    259 Views
    bleaderB
    Sorry for the delay, I'm a bit swamped. That does not ring a bell to me right now. What is you setup like? How many pools, how many host per pool, is there bond on some of them? Then, more to debug what was actually created as you stated there network does exist but there is no traffic: On your hosts: xe network-list to get the uuid of one of these private networks you created xe network-param-list uuid=<netwok-uuid> should tell you in which bridge they are ovs-vsctl show shows all bridges and their ports, in there you should be seeing the bridge you found in previous step. This bridge should have: a port with type vxlan and options in which you have a remote_ip to the network center the VIFs for the VMs On your VM: is that network assigned to the VMs? do the VM have new devices created in ip link or similar when you attached the network? is there any error in VMs dmesg on device creation? We'll see from there if we can get an idea of what is happening.
  • J

    Issue Importing Dell PPDM OVA into XOA

    Unsolved
    11
    2
    0 Votes
    11 Posts
    991 Views
    AtaxyaNetworkA
    Edit to my previous post: the script is checking for working DNS, so i cannot pre-setup network. If you want the fixed applicance with the drivers installed, ping me ! To setup the network, you just need to login (root/changeme, yeah the password is... bleh), and launch the script /usr/local/brs/puppet/scripts/install.sh -lui