Hi @olivierlambert,

No performance gain to keep the snapshot. It's more the opposite. Leaving the snapshot means it will be only removed in the next job, meaning you'll have more to coalesce. However, it's very stable, CBT tends to have some quirks from time to time, so worth trying on few VMs first.

Thanks - perhaps the backup documentation could have a page that covers each of these options and the pro's / con's of enabling / disabling each of them.
They seem to be newer options added as they weren't even shown in the webinar video and I couldn't find any mention of them anywhere.

Also, if it's better to have this option enabled, then perhaps it could be enabled by default ?

I don't think it's possible now. I can ask internally if it's something that's planned already or doable in the future

Thanks - that'll be good. Either option will be good. The ability to set the 'Full Backup Interval' against the individual schedule instead of the whole backup task may be all that needs to change.
I'll submit a ticket to get this feature request started.

Thanks
Kent.

I've been unable to find documentation on the toggle options when setting up a backup schedule (I've watched the video and read the documentation a couple of times - but acknowledge it could be documented and I've read over it / completely missed it.

Note:
• I'm backing up using continuous replication to another XCP server over a private LAN connection in another city, so bandwidth is less than local backups.
• It took 2.5 days to complete the first full backup
• I have (currently) 168 VMs to back up, all with drives between 80 and 500GB in size. Total backup size is 15TiB.
• My 15TiB has somehow ballooned and completely filled the 32TiB drive (I've now resized it to 40TiB).
• I'm using 'Use NBD + CBT to transfer disk if available' for backups.

Questions:

1. Purge snapshot data when using CBT
   I note that there is also a 'base copy' that I assume has been created as well. I'm assuming that this is used for the CBT tracking between each backup.
   The tooltip says they won't use any notable space on disk, not visible in the UI and cannot be restored from.

Question: Are there any performance gains for keeping the last backup snapshot between backups ?

2. Force full Backup
   I am doing a Mon-Fri backup (keep 6) and a Sat Backup (keep 4).
   I currently have the Sat backup set to 'Force Full Backup'

With 168+ VM's, I'd like to instead rotate the Full backup so that only a subset (say 40) are done each Saturday on a rotating basis.
I did look at doing a full backup on the first Saturday of each month, but this doesn't seem possible either.
I also considered doing the backups by physical host, but you can't use the smart filters to a host, just a pool.

Question: Any way to stagger the Full Backup of VM's within a schedule ?

Thanks
Kent.

@olivierlambert

Dou! Thanks for clarifying the speed limit - I've adjust to 100MiB, which should use about 80% of my available bandwidth.

Resolving the VDIs' appears to have worked as backups going now.

I just have these long running tasks which have been going multiple days with no end in sight that I can't resolve. I suspect they may be related to why I can't migrate VM's between hosts now.

Cheers
Kent.

@olivierlambert

VDI in use is something else, it means the VDI is still mounted somewhere. It's very hard to tell you exactly where, maybe stuck attached to the Dom0.

I tracked down these - in once case the same VDI was linked to three different hosts. Restarting the toolstack on the master and these hosts didn't release the vdi, in the end the only way I was able to do so was to restart the physical host. I ended up having to do this on 7 of my 12 separate hosts to release all the VM's that were in this state.

The latest issue I'm having is I can't migrate VM's between different pools since I enabled backups (at least I believe it's since I enabled the backups). I do not that there are jobs still running after 2+ days and say they have days left to run. Again, I've restarted the toolstack and physical host (at least twice) of MHP101 but they keep coming back after the restart. It's been over a day since I last restarted and they appear stuck.

What's going on here and how do I get rid of these tasks ?

Cheers
Kent.

@olivierlambert

Another issue I have, is I have two VM's which fail to back up despite multiple 'full' attempts. After a couple of failed attempts, I have deleted the snapshot that's left behind, the backup VM that's created (basically anything that has the VM name in it other than the original machine itself) and tried again (twice).
The other VM's in this schedule all backed up fine.

What am I looking for to resolve this issue ?

UUID: ccb53826-eb4a-4b42-b397-2882ad92e17f
Start time: Monday, January 6th 2025, 9:35:17 pm
End time: Monday, January 6th 2025, 10:33:56 pm
Duration: an hour
️ Transfer data using NBD
Error: VDI_IN_USE(OpaqueRef:90cc2a1e-4480-318c-a791-32536f9aad02, destroy)

UUID: a826a1d8-9bc1-87a0-0225-f5e7c27c4d84
Start time: Monday, January 6th 2025, 9:40:13 pm
End time: Monday, January 6th 2025, 10:48:00 pm
Duration: an hour
️ Transfer data using NBD
Error: VDI_IN_USE(OpaqueRef:61fc3b2b-fe6b-a6e0-a9ed-5f1f18676b98, destroy)

Transfer data using NBD
Snapshot
Start: 2025-01-06 21:35
End: 2025-01-06 21:35
Local storage (179.85 TiB free - thin) - MHP101
transfer
Start: 2025-01-06 21:36
End: 2025-01-06 22:14
Duration: 38 minutes
Size: 74 GiB
Speed: 32.92 MiB/s
Start: 2025-01-06 21:35
End: 2025-01-06 22:14
Duration: 39 minutes

Start: 2025-01-06 21:35
End: 2025-01-06 22:33
Duration: an hour
Error: VDI_IN_USE(OpaqueRef:90cc2a1e-4480-318c-a791-32536f9aad02, destroy)This is a XenServer/XCP-ng error
Type: full

@olivierlambert

Thanks - the other question I have is regarding the "Full backup interval" option.

The documentation suggests doing this periodically (eg. 20) in case the backup Delta itself gets corrupt.

I have (currently) 168 servers totalling ~65TB storage. I'm using the [NOBAK] option on OS and local backups, reducing this down to ~22Tb that I'm using Continuous Replication on.

I'm trying to figure out the best way to spread this so that only say 10 servers are reset each time on a rotating basis.

Open to ideas ?

Thanks
Kent.

Hi,

We have a 1GiB backhaul connection between our DC and office, which is where I have our backup server (running XMP8.3).

So that we don't flood our available bandwidth, I've enabled a 'Speed limit' of 800MiB/s, which should be well below our 1000MiB/s backhaul, however I see our connection is at capacity and our provider is reporting they are seeing significant dropped packets occurring on our backhaul.

This is how I've configured the backup schedule:

Concurrency: 6
NBD connections: 4
Purge snapshot data when using CBT: Enabled
Full backup interval: 8
Speed limit: 800 MiB/s

So my question is, is the 'Speed limit' intended to be overall, per VM or per NDB connection ?

@olivierlambert

Thanks, I had read this guide earlier but hadn't processed it properly.

XenCenter: star hamil bream

XOA: |(star hamil bream)

I've set up a saved filter as |( ) and made this my default, so hopefully this will get me to what I'm used to.

Cheers
Kent.

Hi,

I'm transitioning from XenCenter to using XOA (I know, should have done years ago, but ...).

Anyway, In XenCenter, I'm used to being able to type part of the servers name into the filter, space, and then part of another servers name, etc. This then would just show me those two+ servers in the list.

I haven't figured out how to do this in XOA. I can type part of the first servers and it shows, but then typing the second returns.

ie. it's doing an 'AND' filter and I want it to do an 'OR' filter.

Am I missing something obvious ?

thanks
Kent.

Hi @olivierlambert ,

Okay, I made some progress today and have worked out the steps to remove, delete or add drives to a VM.

Removed drive from the server attached to and also deleted the disk itself

xo-cli vdi.delete id=<the id of the VDI>

Removed drive from the server attached to, but didn't delete the disk!

xo-cli vdb.delete id=<the id of the VBD>

Added a drive to the server

xo-cli vm.attachDisk vdi=<the id of the VDI> vm=<the id of VM>

If was the last one that I was missing. I'm still not sure of the difference between the 'uuid' and 'id' fields, but will use the 'id' as this seems to be what the commands are wanting.

Thanks
Kent.

Hi @olivierlambert ,

Okay - I spent some time today and have my head around the cli basics. So far I've worked out how to read the object list to get the VM's, VDI's and VBD's.

I now exploring around automating my build process.

I've worked out how to clone the VM, then get the VBDs and VDI's of the drives attached to that VM.
I can even see via the xo-cli that I can delete the VM and keep the drive.

Questions:
2) Most items have both a 'uuid' and an 'id', but as far as I can tell, they are always the same. Is there any difference ?

2. What is the best way to remove the existing OS drive and add the newly cloned drive ? (I wonder if I can just edit the VBD record to change the VDI it points to, but I think I'm missing a step in the process.

3. Is this documented and I've just not found it yet. The link above gives the same as the xo-cli does. and the xo-cli list-commands gives the commands, but minimal documentation on how to use.

thanks
Kent.

Is there a way to clone a drive without having to also clone the whole VM ?

Also, is there a way (via the XO interface) to delete the VM and not delete the drive assigned to it like is possible with XenCenter ?

It's fast (individual server down time is ~5 minutes), it's been a rock solid reliable process in all the years I've been doing it, plus it's relatively simple to do. It also allows doing rollouts like what I did in the weekend, which was a clean install of Windows Server 2022 and a major update to our DB software.
There are also some aspects of the DB updating process which are less than reliable.

Hi All,

API Question: Can I automate this process a bit more ?

1. Use the API to clone (fast) a drive (not the whole VM, just a drive) ?
2. Use the API to shut down a VM, delete the OS drive, install the previously cloned drive, start the VM up again ?

Background: I have 150+ VM's where they have a multi drive setup:

1. OS Drive. This is based on my template OS. Only the server name, IP address and DB Server config is different
2. Data, Storage and Backup Drives. These three are unique to each instance.

To roll out OS and/or DB updates, I have a process where I:

1. Prepare the template OS.
2. Clone the template VM (normally in batches of 10-20 at a time)
3. Delete the cloned templates, keeping the OS drive.
4. For each of the 150+ VM's
   i. Shut it down
   ii. Delete the OS drive
   iii. Add one of the cloned OS drives
   iv. rename the drive name as appropriate
   v. start up the VM
5. On first start, the new OS drive has a script which runs to automatically set the server name, IP address and DB Server config (each VM's unique settings are saved in a config file on one of the other drives which my config script reads from).

I've been using this process for a number of years, 3 or 4 times a year - previously with the Windows XenCenter. This time round I tried using XO instead, but there was one step that was painful.
On XenCenter, after making 20x clones of the template. I can select all 20 and delete in one go. Before the actual delete, I can select to NOT delete the OS drive.
With XO, the only way I could figure out how to do this was individually editing each clone to disconnect the drive.

The other difference is, with XenCenter the original copy was always listed last. With XO, the original copy appeared to be in the middle of the list and more than once I ended up adding the original copy to a machine by mistake and only discovering when I went to make more clones and discovering the drive name was renamed to one of the VM's I'd just updated. I worked around this by renaming it after doing the clones, then renaming it back before creating more clones.

Cheers
Kent.

@AtaxyaNetwork said in User cannot see any servers - how to edit permissions:

https://xen-orchestra.com/blog/warm-migration-with-xen-orchestra/

Thanks for this - I wasn't aware of this option. However, it's dimmed out for me and I see that this is an 'Enterprise' only feature.

@AtaxyaNetwork

Thanks, I'd come to this conclusion after posting. I must have done this when I was using a free trial as it was all working until the weekend when we updated hardware and now have new pools - so he can't see the new pools (and the old pools have gone). I did find an error in the logs about permissions, so this all ties up.

Followup question: Do you have Xen Orchestra just running as a VM in the pool or on it's own server ?

Reason: Due to CPU hardware differences, I wasn't able to do a live migration between our old servers and new. This meant shutting down each VM, moving and then starting up again. I obviously couldn't do this with the VM running Xen Orchestra and used XCP-ng Centre app (confession, I ended up doing ~90% of moves via the app. I'd tried using Xen Orchestra but the extra clicks involved alone made an already laborious task more so).

I've set up Xen Orchestra with myself as Admin and one of my staff with 'user'.

Today when they logged on, they couldn't see any servers. When I changed them to 'Admin' all VM's showed.

Where do I control what VM's the 'User' account can see ?

Restarting the toolstack on AHP101 (which was the pool master) cleared the tasks.

Hopefully no long term side effects

@Danp
They are still here the next morning. I've already tried to restart XOA and also still there.

I guess next option is to restart the toolstack on AHP101.

Cheers
Kent.

I've been migrating VM's from old servers to new servers and notice that I now have 34+ tasks that are on 0% and some of them are going 5+ hours later.

What are they? And should I be worried? And what can I do to clear them up ?

Thanks
Kent.