Best posts made by cg

cg

@stormi said in XCP-ng 8.0.0 Beta now available!:

You could test with Citrix Hypervisor and then report it at https://bugs.xenserver.org
Maybe they'll want to investigate even if the hardware is not officially supported. If someone can reproduce the issue on supported hardware, that would make things easier.

Though I doubt it was woth the effort, as the support over there is well known (especially for not supported configurations etc.) I did it - for the reason of trying to improve stuff (and giving back to OSS community):
https://bugs.xenserver.org/projects/XSO/issues/XSO-955

cg

@stormi dd stands for disk dump and does exactly that: Copy a stream of data.
Fio however can be configured for precise workloads and read/write mixes, parallel workloads etc.

So the first thing will only give you streamline benchmarks, what almost nobody cares about.
The second can simulate realworld (VM/database...) workloads, where (controller) Caches and non magnetic storage (Flash, Optane, MRAM...) makes the real difference.
Also use big amount of data, since caches can impact small ones extremely. Don't get me wrong: We need them and they can make huge differences, but as long as your benchmarks fully fit into them, it gives your nonsense/fake results. Also (consumer) SSDs start throttling after some 10 to a very few 100 GB of data written. Their caches fill up and they 'overheat'.

You can spend days on benchmarks and how to do what.

cg

The problem is still persistent or exisiting again.

Exactly: You can use ALT GR once and get the symbol you need (@ |...) but afterwarts it's just putting weird symbols into the console.
Dirty fix: Change the tab to whatever (general, stats...) and and back to console.

Still pretty annoying and kicked me out of one of my servers for some time (thx fail2ban...).

cg

https://xen-orchestra.com/blog/xen-orchestra-5-36/

The part about ZFS:

As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create your SR.

Check this documentation for more information about ZFS SR creation.

Once the SR is created, you will have the option to select it during the SR creation directly in Xen Orchestra.

Since it's confusing and technically wrong, it should rather be something like:

As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create a ZFS pool before creating the corresponding SR.

Check this documentation for more information about ZFS SR creation.

Once the ZFS pool is created and mounted, you will have the option to select the mountpoint during the SR creation directly in Xen Orchestra.

...and this forum is missing a "quote" button, as formating text in codeboxes doesn't work (and tags don't survive a linebreak, you have to mark it every line again).

Regards,
Christof

cg

@bleader

IIRC I just "tried again".
It failed 2 times, then I looked up the logs from other console, removed the file (which shouldn't be of any importance for our instance) and retried without reboot.

I copied the whole installer-log to the usb stick before finshing the install.
(Could actually be a good hint or even a menu-option for those, where the install fails and won't leave it on the harddrive - e.g. evaluating hardware)

[  128.517356] ata1.00: exception Emask 0x0 SAct 0x800000 SErr 0x0 action 0x0
[  128.517357] ata1.00: irq_stat 0x40000008
[  128.517359] ata1.00: failed command: READ FPDMA QUEUED
[  128.517362] ata1.00: cmd 60/80:b8:10:6c:d4/00:00:02:00:00/40 tag 23 ncq dma 65536 in
         res 41/40:10:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.517363] ata1.00: status: { DRDY ERR }
[  128.517364] ata1.00: error: { UNC }
[  128.518008] ata1.00: configured for UDMA/133
[  128.518018] sd 0:0:0:0: [sda] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.518020] sd 0:0:0:0: [sda] tag#23 Sense Key : Medium Error [current] 
[  128.518021] sd 0:0:0:0: [sda] tag#23 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.518024] sd 0:0:0:0: [sda] tag#23 CDB: Read(10) 28 00 02 d4 6c 10 00 00 80 00
[  128.518025] print_req_error: I/O error, dev sda, sector 47475840
[  128.518039] ata1: EH complete
[  128.581286] ata1.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x0
[  128.581287] ata1.00: irq_stat 0x40000008
[  128.581288] ata1.00: failed command: READ FPDMA QUEUED
[  128.581291] ata1.00: cmd 60/08:c8:80:6c:d4/00:00:02:00:00/40 tag 25 ncq dma 4096 in
         res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.581292] ata1.00: status: { DRDY ERR }
[  128.581293] ata1.00: error: { UNC }
[  128.582111] ata1.00: configured for UDMA/133
[  128.582117] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.582118] sd 0:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] 
[  128.582119] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.582121] sd 0:0:0:0: [sda] tag#25 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
[  128.582122] print_req_error: I/O error, dev sda, sector 47475840
[  128.582133] ata1: EH complete
[  128.629307] ata1.00: exception Emask 0x0 SAct 0x200 SErr 0x0 action 0x0
[  128.629309] ata1.00: irq_stat 0x40000008
[  128.629310] ata1.00: failed command: READ FPDMA QUEUED
[  128.629313] ata1.00: cmd 60/08:48:80:6c:d4/00:00:02:00:00/40 tag 9 ncq dma 4096 in
         res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.629314] ata1.00: status: { DRDY ERR }
[  128.629315] ata1.00: error: { UNC }
[  128.630068] ata1.00: configured for UDMA/133
[  128.630074] sd 0:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.630076] sd 0:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] 
[  128.630077] sd 0:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.630078] sd 0:0:0:0: [sda] tag#9 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
[  128.630079] print_req_error: I/O error, dev sda, sector 47475840
[  128.630092] ata1: EH complete

Indeed it looks like the SSD should be replaced.

8.3 is running stable on this (and all other hosts, I upgraded) so far.
It's a system at a UAS, running various student projects for several years now, coming from XenServer originally. I voluntarily maintain it. Thx for the hint!

cg

@dinhngtu I can't say on XCP-ng side, but it's likely linked to:
August patch (and following), as Microsoft changed something to the NVMe stack.

e.g.
https://learn.microsoft.com/en-us/answers/questions/5536733/potential-ssd-detection-bug-in-windows-11-24h2-fol

Google gives a lot about it. It seems that it most likely doesn't kill NVMes but can cause trouble.
We have a few PCs becoming more unstable (BSODs) or even very slow after that upgrae.

cg

@olivierlambert
Okay, so bascially what I (and other people, I know, who are in a similar environment) need:

Tape backup is mandatory, as we're talking about (deduplicated, compressed) dozens of TB
Deduplication and good compression: duplicated, uncompressed ends in PB
All of that needs to scale! E.g. Commvault scales on 16 threads here even with only 1 - 2 tasks. (Can't say if even more without upgrade of CPU)
(We need chains, of course, to go B2D2T -> dumping the dedup-store on tape as disaster recovery)
Applicationawareness (which can be done via agent - agentless is not always king or very important):
-- MS Exchange, recovering datastores, mailboxes and even single mail items
-- SQL Servers: MS SQL, MySQL (MariaDB)...
-- Windows Active Directory Items

Over here I also need to have the option to do backups only inside the VM via agent, as I can't snapshot them.
You might have gotton me wrong: AFAIK quiesced backups always made a VM snapshot and submitted the request also to the VSS writer and then the VM/VHDs have been "read". In this case it will always fail, as there's no space on storage for such. Also Citrix discontinued that in their VM tools! XCP-ng yet lacks a proper tool maintenance with corresponding releases, which would be the only chance to keep that.

As an idea for XOA:

Make it run on physical hardware, as VMs are too small for such
You already cooperate with HPE - let them run bare metal on ProLiants!
ZFS(oL) Could be your way to go to implement dedup and compression

...feel free to add, comment... whatever.

cg

Citrix also has a long history of problematic VM guest-tools. Race conditions with broken drivers, unbootable VMs, broken agents... It's a pain in the a**.
If you like your VMs: Always make a snapshot of every VM before updating the tools.

Is known why they remove the VSS-functionality? I don't get why they remove features (again).

cg

XCP-ng already ate a bunch of time, and I'm sure it'll continue for testing etc
I'm Sysadmin with very limited coding ability - I'm definately out for that.

I guess it would need someone being good at C#.NET. Also XAPI would probably need to know about it (what may already be, as you're implementing it into XOA)...

cg

First rule of all benchmarks: The longer and more often they run, the more precise they are.
If we talk about 1G as base, why should I switch with 10 or 100 to M? That doesn't make any sense at all.

cg

@DustinB doesn't it use the exact same mechanism?
I have to find out.

cg

I had that many times with XCP-ng 8.2 and it felt like the problem is gone with 8.3, but it seems it's not (completely) gone.

I can enable passthrough, add it to VM and it runs fine.
I shutdown the VM for whatever reasons, recently it was just a reboot for Windows updates and the USB device got lost.

What could be the reason, how to debug that?

I can enable it again, shutdown VM (hotplug sadly still doesn't work till today?) attach and it's fine again.

Regards

Christof

cg

As I can't find it anywhere:
What exactly is happening on join/leave?
Is the masking changing at runtime and affecting every new started VM, does the pool need to be rebooted after an older server has been removed, to "unlock" all features?

I have a pool with HPE DL380 Gen 9 and 10 where Gen 9 will replaced with Gen 12.

So I assume it makes sense to remove 9 first, then add 12.
But how exactly is the masking process working?
Should I reboot Gen 10 first (shutting down all VMs) or add 12, move VMs over... or just shutdown and start all currently running VMs?

In the early days (~XenServer 6) it had to be done manually by getting CPU features, calculating the commons and give it as parameter to join command.
What I did not see though, is: How exactly is it working on removal and what's best practice.

(also: should be added to documentations)

cg

@AtaxyaNetwork did you ever think about "upgrading" the system?
The installer should be able to detect and backup your current one, installl new and migrate all relevant data.
That should exclude whatever home-brew-fiddlydiddly you did to your current installation.
The reason you're seeing 2 almost identical partitions is exactly that: The way how upgrades work with XenServer/XCP-ng. One is your current system, the other one may contain a backup of a previous install (which can also be restored by the installer - if existing).

As it's been mentioned: Neither trust USB-storage nor any AI without knowing what they suggest you to do. But you may have learned your AI-lesson: It can help, but it can end with a big pile of sh*t if you don't know what it's suggesting you. It's more A than I.

cg

Small warning: XenTools are still a clusterfuck.
On (allmost) all machines without our main environment it keeps the ip-configuration within the Realtek emulated NIC and never transfers it to the Xen device.
It's fucked for long now - but doesn't matter if all VMs are getting their IPs via DHCP.

Currently struggling with: Windows Server 2016 + XenTools/Management 9.4.2.
At least it didn't break the whole VM by destroying the disk device drivers this round.

tl;dr: Snapshot your VMs and have a documentation at hand, when upgrading the tools/drivers.

cg

@bikemuch I registered an XOA-account to keep a few machines with XOA maintained. It's free, not much to loose.

Many of us are looking forward for a 9.0 release, including a modern 5.x or even 6.x kernel, including all the developments and improvements, that have been made.
Also things like 16TB+ disks on block storage (incl. thin provisioning) etc.

If your company is running a bigger environment: You can put priorities on features as a paying customer.

cg

@bikemuch While you're technically not wrong about the impression, especially Linux distros usually work with flawlessly with the former version template.
I also saw that there's a user-test running with updated templates including Debian 13 etc.
It should be GA in about 2 weeks.

cg