@Biggen At the moment, xcp-ng center provides some better views and overviews not yet available in XO.. Hoping next major version fixes this
Best posts made by Forza
-
RE: [WARNING] XCP-ng Center shows wrong CITRIX updates for XCP-ng Servers - DO NOT APPLY - Fix released
-
RE: Best CPU performance settings for HP DL325/AMD EPYC servers?
Sorry for spamming the thread.
I have two identical servers (srv01 and srv02) with AMD EPYC 7402P 24 Core CPUs. On srv02 I enabled the
LLC as NUMA Node
.I've done some quick benchmarks with
Sysbench
on Ubuntu 20.10 with 12 assigned cores. Command line:sysbench cpu run --threads=12
It would seem that in this test the NUMA option is much faster, 194187 events vs 103769 events. Perhaps I am misunderstanding how sysbench works?
With 7-zip the gain is much less, but still meaningful. A little slower in single-threaded performance but quite a bit faster in multi-threaded mode.
-
RE: Host stuck in booting state.
Problem was a stale connection with the NFS server. A reboot of the NFS server fixed the issue.
-
RE: Restoring a downed host ISNT easy
@xcprocks said in Restoring a downed host ISNT easy:
So, we had a host go down (OS drive failure). No big deal right? According to instructions, just reinstall XCP on a new drive, jump over into XOA and do a metadata restore.
Well, not quite.
First during installation, you really really must not select any of the disks to create an SR as you could potentially wipe out an SR.
Second, you have to do the sr-probe and sr-introduce and pbd-create and pbd-plug to get the SRs back.
Third, you then have to use XOA to restore the metadata which according to the directions is pretty simple looking. According to: https://xen-orchestra.com/docs/metadata_backup.html#performing-a-restore
"To restore one, simply click the blue restore arrow, choose a backup date to restore, and click OK:"
But this isn't quite true. When we did it, the restore threw an error:
"message": "no such object d7b6f090-cd68-9dec-2e00-803fc90c3593",
"name": "XoError",Panic mode sets in... It can't find the metadata? We try an earlier backup. Same error. We check the backup NFS share--no its there alright.
After a couple of hours scouring the internet and not finding anything, it dawns on us... The object XOA is looking for is the OLD server not a backup directory. It is looking for the server that died and no longer exists. The problem is, when you install the new server, it gets a new ID. But the restore program is looking for the ID of the dead server.
But how do you tell XOA, to copy the metadata over to the new server? It assumes that you want to restore it over an existing server. It does not provide a drop down list to pick where to deploy it.
In an act of desperation, we copied the backup directory to a new location and named it with the ID number of the newly recreated server. Now XOA could restore the metadata and we were able to recover the VMs in the SRs without issue.
This long story is really just a way to highlight the need for better host backup in three ways:
A) The first idea would be to create better instructions. It ain't nowhere as easy as the documentation says it is and it's easy to mess up the first step so bad that you can wipe out the contents of an SR. The documentation should spell this out.
B) The second idea is to add to the metadata backup something that reads the states of SR to PBD mappings and provides/saves a script to restore them. This would ease a lot of the difficulty in the actual restoring of a failed OS after a new OS can be installed.
C) The third idea is provide a dropdown during the restoration of the metadata that allows the user to target a particular machine for the restore operation instead of blindly assuming you want to restore it over a machine that is dead and gone.
I hope this helps out the next person trying to bring a host back from the dead, and I hope it also helps make XOA a better product.
Thanks for a good description of the restore process.
I was wary of the metadata-backup option. It sounds simple and good to have, but as you said it is in no way a comprehensive restore of a pool.
I'd like to add my own oppinion here. A full pool restore, including network, re-attaching SRs and everything else that is needed to quickly get back up and running. Also a restore pool backup should be available on the boot media. It could look for a NFS/CIFS mount or a USB disk with the backup files on. This would avoid things like issues with bonded networks not working.
-
RE: Remove VUSB as part of job
Might a different solution be to use a USB network bridge instead of direct attached USB? Something like this https://www.seh-technology.com/products/usb-deviceserver/utnserver-pro.html (There are different options available)... We use my-utn-50a with hardware USB keys and it has shown to be very reliable over the years.
-
RE: Netdata package is now available in XCP-ng
@andrewm4894 said in Netdata package is now available in XCP-ng:
Qq, what would be the best way for me to try spin up a sort of test or dev XCP-ng env for me to try things out on? Or is there sort of hardware involved such that this might not be so easy. In my mind I'm imagining spinning up a VM lol which probably shows my level of naivety
You can run XCP-ng inside a VM, as long as the hypervisor underneath exposes nested virtualisation. The actual installation of XCP-ng is very easy. Mostly click and run.
-
RE: [DEPRECATED] SMAPIv3 - Feedback & Bug reports
@olivierlambert hi. I'm also eager to see how the new v3 is progressing. From my company point of view, being able to compact VDIs using guest trim/unmap is very valuable as it minimises storage space usage and improves backup/restore speeds.
-
RE: IP Address changed for a slave within a Pool, How do I reconfigure it?
For now, a warning in XOA when changing the management IP would be nice. It could include an instruction on how to update the pool members after the IP of the master was changed.
Is it possible to use hostnames instead of IPs? It could make things easier?
-
RE: IP Address changed for a slave within a Pool, How do I reconfigure it?
@olivierlambert said in IP Address changed for a slave within a Pool, How do I reconfigure it?:
On a slave host, the database is in read only. If the slave lost the connection with the master, there's no way to make any change into the "local" slave XAPI database.
It makes sense. It would have been nice if pool members get the updated IP through XAPI before the actual change on the master happens.
Latest posts made by Forza
-
RE: VMβs Slow booting
@bassopt said in VMβs Slow booting:
@olivierlambert hi, again!
Nah, as told before itβs on initramdisk. After loading grub. A Ubuntu test install I did freezes for almost 20 seconds there.
I do have some warnings when I boot xcp-ng regarding x-273 and x-297 .... would that cripple performance that much. ( consumer hardware )
Sounds like you have the wrong template. If you disable "quiet" option in grub (press e, then go to the kernel command line) you may see it is error out on enabling some hardware feature. I have seen it happen when trying to enumerate the CPUs. Have a look. You might be able to see in
dmesg
orjournalctl
too. -
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
I did a 3 x backup of a single VM hosted on an SSD drive on the same host as XOA is running, which is also pool master.
This is with
nconnect=1
:
This is with
nconnect=16
:
The transfer speed according to XOA is slightly less, but looking at the bandwidth graph, it looks like the LACP bonded network on the storage server is reaching a higher max throughput.
I will test some more with incremental backups and see if there's a difference with them.
If we ignore the
nconnect
for a second, and just look at the graphs, it seems we have a lot of possibilities to improve the backup performance if we could make the transfer more even. What is causing this type of pattern? I do not believe any coalesce was happening during this test. -
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
Ah, that's a shame, but reasonable for a point release. Maybe next major release?
This was an interesting read regarding nconnect on Azure https://learn.microsoft.com/en-us/azure/storage/files/nfs-performance
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
Changing nconnect requires restart of XOA (seems remount doesn't take a new value for nconnect), and I am heading home now, so I will try some additional benchmarks later in the week.
Btw, does xcp-ng 8.3 support nconnect? 8.2 that I am using does not.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
That would explain why the transfer rate is uneven, as some bandwidth and IO is used by the coalescing processes.
Overall the transfer is somewhat faster. about 1.5 hours for 253GiB on 1Gbit/s connection. Before it was maybe 2ish hours.
I lost the backup logs in XOA since I switched to the new version, but looking back in the Netdata stats from the backup server we can see the following pattern for the previous run of the same backup job. Just looking at the timestamps it looks pretty similar, while the bandwidth used looks less. Maybe some rounding errors in Netdata graphs?
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
@olivierlambert
Yes, I am checking on the XOA VM itself.Here is a view over the XOA VM. You can see that the full backup was started around 3:10 pm:
This is the NFS stats from the backup server (Remote) for the same period:
Why is the server doing reads? Maybe coalesce? But is there coalesce on full backups?
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
Testing full backups. Seems at least XOA is saturating the link for larger VMs. It does have quite a bit of delay outside of the transfer itself, so the overall BW that is logged in the web interface is lower.
This is
bmon
running inside XOA.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
I usually read the blog posts, but I must have missed it that time.
As for performance it is a little difficult to test i a structured manner, but the initial incremental backup seems to run very well. Normally I have about 60-90MB/s as max before, but also often around 20-30MB/s. This time the backup took 6 minutes, and normal is around 15 minutes. This is over a 1 Gbit/s network.
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
I will do.
It would be nice to have an notice that a new XOA is available, in addition to the normal updates. Perhaps something on the https://xoa/#/xoa/support page?
-
RE: Updated XOA with kernel >5.3 to support nconnect nfs option
@olivierlambert said in Updated XOA with kernel >5.3 to support nconnect nfs option:
XOA is based on Debian 12. I'm not against doing a custom kernel if it's really a big bonus in terms of performance
Debian 12 should use kernel 6.1, so that would be ok. Maybe I am having an old XOA as my kernel is 4.19. This is what shows when I login:
login as: xoa xoa@<redacted>'s password: Linux <redacted> 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64 __ __ ____ _ _ \ \ / / / __ \ | | | | \ V / ___ _ __ | | | |_ __ ___| |__ ___ ___| |_ _ __ __ _ > < / _ \ '_ \ | | | | '__/ __| '_ \ / _ \/ __| __| '__/ _` | / . \ __/ | | | | |__| | | | (__| | | | __/\__ \ |_| | | (_| | /_/ \_\___|_| |_| \____/|_| \___|_| |_|\___||___/\__|_| \__,_| Welcome to XOA Unified Edition, with Pro Support. * Restart XO: sudo systemctl restart xo-server.service * Display status: sudo systemctl status xo-server.service * Display logs: sudo journalctl -u xo-server.service * Register your XOA: sudo xoa-updater --register * Update your XOA: sudo xoa-updater --upgrade OFFICIAL XOA DOCUMENTATION HERE: https://xen-orchestra.com/docs/xoa.html Support available at https://xen-orchestra.com/#!/member/support In case of issues, use `xoa check` for a quick health check. Build number: 21.01.02 Based on Debian GNU/Linux 10 (Stable) 64bits in PVHVM mode