@stormi said in XCP-ng 8.0.0 Beta now available!:

You could test with Citrix Hypervisor and then report it at https://bugs.xenserver.org
Maybe they'll want to investigate even if the hardware is not officially supported. If someone can reproduce the issue on supported hardware, that would make things easier.

Though I doubt it was woth the effort, as the support over there is well known (especially for not supported configurations etc.) I did it - for the reason of trying to improve stuff (and giving back to OSS community):
https://bugs.xenserver.org/projects/XSO/issues/XSO-955

@stormi dd stands for disk dump and does exactly that: Copy a stream of data.
Fio however can be configured for precise workloads and read/write mixes, parallel workloads etc.

So the first thing will only give you streamline benchmarks, what almost nobody cares about.
The second can simulate realworld (VM/database...) workloads, where (controller) Caches and non magnetic storage (Flash, Optane, MRAM...) makes the real difference.
Also use big amount of data, since caches can impact small ones extremely. Don't get me wrong: We need them and they can make huge differences, but as long as your benchmarks fully fit into them, it gives your nonsense/fake results. Also (consumer) SSDs start throttling after some 10 to a very few 100 GB of data written. Their caches fill up and they 'overheat'.

You can spend days on benchmarks and how to do what.

The problem is still persistent or exisiting again.

Exactly: You can use ALT GR once and get the symbol you need (@ |...) but afterwarts it's just putting weird symbols into the console.
Dirty fix: Change the tab to whatever (general, stats...) and and back to console.

Still pretty annoying and kicked me out of one of my servers for some time (thx fail2ban...).

https://xen-orchestra.com/blog/xen-orchestra-5-36/

The part about ZFS:

As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create your SR.

Check this documentation for more information about ZFS SR creation.

Once the SR is created, you will have the option to select it during the SR creation directly in Xen Orchestra.

Since it's confusing and technically wrong, it should rather be something like:

As there are many possibilities during the creation of a ZFS storage repository, you still need to use the command line (eg. zpool create zdata /dev/sdb) to create a ZFS pool before creating the corresponding SR.

Check this documentation for more information about ZFS SR creation.

Once the ZFS pool is created and mounted, you will have the option to select the mountpoint during the SR creation directly in Xen Orchestra.

...and this forum is missing a "quote" button, as formating text in codeboxes doesn't work (and tags don't survive a linebreak, you have to mark it every line again).

Regards,
Christof

@bleader

IIRC I just "tried again".
It failed 2 times, then I looked up the logs from other console, removed the file (which shouldn't be of any importance for our instance) and retried without reboot.

I copied the whole installer-log to the usb stick before finshing the install.
(Could actually be a good hint or even a menu-option for those, where the install fails and won't leave it on the harddrive - e.g. evaluating hardware)

[  128.517356] ata1.00: exception Emask 0x0 SAct 0x800000 SErr 0x0 action 0x0
[  128.517357] ata1.00: irq_stat 0x40000008
[  128.517359] ata1.00: failed command: READ FPDMA QUEUED
[  128.517362] ata1.00: cmd 60/80:b8:10:6c:d4/00:00:02:00:00/40 tag 23 ncq dma 65536 in
         res 41/40:10:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.517363] ata1.00: status: { DRDY ERR }
[  128.517364] ata1.00: error: { UNC }
[  128.518008] ata1.00: configured for UDMA/133
[  128.518018] sd 0:0:0:0: [sda] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.518020] sd 0:0:0:0: [sda] tag#23 Sense Key : Medium Error [current] 
[  128.518021] sd 0:0:0:0: [sda] tag#23 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.518024] sd 0:0:0:0: [sda] tag#23 CDB: Read(10) 28 00 02 d4 6c 10 00 00 80 00
[  128.518025] print_req_error: I/O error, dev sda, sector 47475840
[  128.518039] ata1: EH complete
[  128.581286] ata1.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x0
[  128.581287] ata1.00: irq_stat 0x40000008
[  128.581288] ata1.00: failed command: READ FPDMA QUEUED
[  128.581291] ata1.00: cmd 60/08:c8:80:6c:d4/00:00:02:00:00/40 tag 25 ncq dma 4096 in
         res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.581292] ata1.00: status: { DRDY ERR }
[  128.581293] ata1.00: error: { UNC }
[  128.582111] ata1.00: configured for UDMA/133
[  128.582117] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.582118] sd 0:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] 
[  128.582119] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.582121] sd 0:0:0:0: [sda] tag#25 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
[  128.582122] print_req_error: I/O error, dev sda, sector 47475840
[  128.582133] ata1: EH complete
[  128.629307] ata1.00: exception Emask 0x0 SAct 0x200 SErr 0x0 action 0x0
[  128.629309] ata1.00: irq_stat 0x40000008
[  128.629310] ata1.00: failed command: READ FPDMA QUEUED
[  128.629313] ata1.00: cmd 60/08:48:80:6c:d4/00:00:02:00:00/40 tag 9 ncq dma 4096 in
         res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.629314] ata1.00: status: { DRDY ERR }
[  128.629315] ata1.00: error: { UNC }
[  128.630068] ata1.00: configured for UDMA/133
[  128.630074] sd 0:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.630076] sd 0:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] 
[  128.630077] sd 0:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.630078] sd 0:0:0:0: [sda] tag#9 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
[  128.630079] print_req_error: I/O error, dev sda, sector 47475840
[  128.630092] ata1: EH complete

Indeed it looks like the SSD should be replaced.

8.3 is running stable on this (and all other hosts, I upgraded) so far.
It's a system at a UAS, running various student projects for several years now, coming from XenServer originally. I voluntarily maintain it. Thx for the hint!

@dinhngtu I can't say on XCP-ng side, but it's likely linked to:
August patch (and following), as Microsoft changed something to the NVMe stack.

e.g.
https://learn.microsoft.com/en-us/answers/questions/5536733/potential-ssd-detection-bug-in-windows-11-24h2-fol

Google gives a lot about it. It seems that it most likely doesn't kill NVMes but can cause trouble.
We have a few PCs becoming more unstable (BSODs) or even very slow after that upgrae.

@olivierlambert
Okay, so bascially what I (and other people, I know, who are in a similar environment) need:

• Tape backup is mandatory, as we're talking about (deduplicated, compressed) dozens of TB
• Deduplication and good compression: duplicated, uncompressed ends in PB
• All of that needs to scale! E.g. Commvault scales on 16 threads here even with only 1 - 2 tasks. (Can't say if even more without upgrade of CPU)
  (We need chains, of course, to go B2D2T -> dumping the dedup-store on tape as disaster recovery)
• Applicationawareness (which can be done via agent - agentless is not always king or very important):
  -- MS Exchange, recovering datastores, mailboxes and even single mail items
  -- SQL Servers: MS SQL, MySQL (MariaDB)...
  -- Windows Active Directory Items

Over here I also need to have the option to do backups only inside the VM via agent, as I can't snapshot them.
You might have gotton me wrong: AFAIK quiesced backups always made a VM snapshot and submitted the request also to the VSS writer and then the VM/VHDs have been "read". In this case it will always fail, as there's no space on storage for such. Also Citrix discontinued that in their VM tools! XCP-ng yet lacks a proper tool maintenance with corresponding releases, which would be the only chance to keep that.

As an idea for XOA:

• Make it run on physical hardware, as VMs are too small for such
• You already cooperate with HPE - let them run bare metal on ProLiants!
• ZFS(oL) Could be your way to go to implement dedup and compression

...feel free to add, comment... whatever.

Citrix also has a long history of problematic VM guest-tools. Race conditions with broken drivers, unbootable VMs, broken agents... It's a pain in the a**.
If you like your VMs: Always make a snapshot of every VM before updating the tools.

Is known why they remove the VSS-functionality? I don't get why they remove features (again).

XCP-ng already ate a bunch of time, and I'm sure it'll continue for testing etc
I'm Sysadmin with very limited coding ability - I'm definately out for that.

I guess it would need someone being good at C#.NET. Also XAPI would probably need to know about it (what may already be, as you're implementing it into XOA)...

First rule of all benchmarks: The longer and more often they run, the more precise they are.
If we talk about 1G as base, why should I switch with 10 or 100 to M? That doesn't make any sense at all.

@bikemuch I registered an XOA-account to keep a few machines with XOA maintained. It's free, not much to loose.

Many of us are looking forward for a 9.0 release, including a modern 5.x or even 6.x kernel, including all the developments and improvements, that have been made.
Also things like 16TB+ disks on block storage (incl. thin provisioning) etc.

If your company is running a bigger environment: You can put priorities on features as a paying customer.

@bikemuch While you're technically not wrong about the impression, especially Linux distros usually work with flawlessly with the former version template.
I also saw that there's a user-test running with updated templates including Debian 13 etc.
It should be GA in about 2 weeks.

@bleader

IIRC I just "tried again".
It failed 2 times, then I looked up the logs from other console, removed the file (which shouldn't be of any importance for our instance) and retried without reboot.

I copied the whole installer-log to the usb stick before finshing the install.
(Could actually be a good hint or even a menu-option for those, where the install fails and won't leave it on the harddrive - e.g. evaluating hardware)

[  128.517356] ata1.00: exception Emask 0x0 SAct 0x800000 SErr 0x0 action 0x0
[  128.517357] ata1.00: irq_stat 0x40000008
[  128.517359] ata1.00: failed command: READ FPDMA QUEUED
[  128.517362] ata1.00: cmd 60/80:b8:10:6c:d4/00:00:02:00:00/40 tag 23 ncq dma 65536 in
         res 41/40:10:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.517363] ata1.00: status: { DRDY ERR }
[  128.517364] ata1.00: error: { UNC }
[  128.518008] ata1.00: configured for UDMA/133
[  128.518018] sd 0:0:0:0: [sda] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.518020] sd 0:0:0:0: [sda] tag#23 Sense Key : Medium Error [current] 
[  128.518021] sd 0:0:0:0: [sda] tag#23 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.518024] sd 0:0:0:0: [sda] tag#23 CDB: Read(10) 28 00 02 d4 6c 10 00 00 80 00
[  128.518025] print_req_error: I/O error, dev sda, sector 47475840
[  128.518039] ata1: EH complete
[  128.581286] ata1.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x0
[  128.581287] ata1.00: irq_stat 0x40000008
[  128.581288] ata1.00: failed command: READ FPDMA QUEUED
[  128.581291] ata1.00: cmd 60/08:c8:80:6c:d4/00:00:02:00:00/40 tag 25 ncq dma 4096 in
         res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.581292] ata1.00: status: { DRDY ERR }
[  128.581293] ata1.00: error: { UNC }
[  128.582111] ata1.00: configured for UDMA/133
[  128.582117] sd 0:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.582118] sd 0:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] 
[  128.582119] sd 0:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.582121] sd 0:0:0:0: [sda] tag#25 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
[  128.582122] print_req_error: I/O error, dev sda, sector 47475840
[  128.582133] ata1: EH complete
[  128.629307] ata1.00: exception Emask 0x0 SAct 0x200 SErr 0x0 action 0x0
[  128.629309] ata1.00: irq_stat 0x40000008
[  128.629310] ata1.00: failed command: READ FPDMA QUEUED
[  128.629313] ata1.00: cmd 60/08:48:80:6c:d4/00:00:02:00:00/40 tag 9 ncq dma 4096 in
         res 41/40:08:80:6c:d4/00:00:02:00:00/00 Emask 0x409 (media error) <F>
[  128.629314] ata1.00: status: { DRDY ERR }
[  128.629315] ata1.00: error: { UNC }
[  128.630068] ata1.00: configured for UDMA/133
[  128.630074] sd 0:0:0:0: [sda] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  128.630076] sd 0:0:0:0: [sda] tag#9 Sense Key : Medium Error [current] 
[  128.630077] sd 0:0:0:0: [sda] tag#9 Add. Sense: Unrecovered read error - auto reallocate failed
[  128.630078] sd 0:0:0:0: [sda] tag#9 CDB: Read(10) 28 00 02 d4 6c 80 00 00 08 00
[  128.630079] print_req_error: I/O error, dev sda, sector 47475840
[  128.630092] ata1: EH complete

Indeed it looks like the SSD should be replaced.

8.3 is running stable on this (and all other hosts, I upgraded) so far.
It's a system at a UAS, running various student projects for several years now, coming from XenServer originally. I voluntarily maintain it. Thx for the hint!

@dinhngtu I can't say on XCP-ng side, but it's likely linked to:
August patch (and following), as Microsoft changed something to the NVMe stack.

e.g.
https://learn.microsoft.com/en-us/answers/questions/5536733/potential-ssd-detection-bug-in-windows-11-24h2-fol

Google gives a lot about it. It seems that it most likely doesn't kill NVMes but can cause trouble.
We have a few PCs becoming more unstable (BSODs) or even very slow after that upgrae.

I often wondered what's the general purpose of that option.
As I only have 1 - 2 socket servers, I always choose 1 socket with x cores (mostly 2 - 8, not exeeding 1 real CPU).
Also for historic reasons: Sockets have been limited, but not cores.

Does it generally make any difference on Xen side/backend?
VM OS might handle it different due to NUMA optimizations.

After upgrading my first server successfully, I upgraded another one recently (different environments, no pool), but it failed.

I remembered from the forum to check installer logs, so I copied the whole directory (in case it contains useful info) and switched to another console (ALT+F3?) to see where it failed.
I don't know if it's documented somewhere, but following that console is pretty informative, rather then just seeing a progress bar.
Older windows types didn't hide what the installer is doing. It's a bit sad XCP-ng "hides" that.

tl;dr: The problem seemed to be:
STANDARD ERROR:

cp: error reading '/tmp/primary-jqbXmQ/usr/lib64/python2.7/lib-dynload/_codecs_hk.so': Input/output error
cp: failed to extend '/tmp/backup-TbutMQ/usr/lib64/python2.7/lib-dynload/_codecs_hk.so': Input/output error

As it was during backup phase, nothing was broken and I could just retry... to end up with the same problem.
As it looks like some hongkong locales, I just removed the file and tried again: with success.
Backup ran through, install/upgrade went fine. Box is running since.

I didn't back the file up, but with "ls" it looked fine like everything else. Also nobody ever touched that file. I can't say why, but wanted to drop it here, for archival purposes. Maybe someone else stumbles over a close or similar problem.

As the logs and other terminal give quite some information about current actions, debugging was somehow fun and it was interesting to dig a bit into what the installer is actually doing. Big pro over Microsoft... which often is a big pain to debug.

If you want the whole installlog-dir: I still have it, but will delete the next days, if not.

Greetings

• Christof

@nikade I found out the HPE MSA2060 has a full flash bundle option, wich is suprisingly cheap, so our SAN has 3.84 TB SAS SSDs - they'll be good within a few hours, but our backup server has a RAID6 with 10 TB HDDs.

@nikade It's also RAS. The risk of a 2nd failing disc during rebuild is a lot higher than usual.
Our B2D2T server needs about 24 hours for that.

@olivierlambert said in First SMAPIv3 driver is available in preview:

Hi,

8.3 release is eating a lot of resources, so that's the opposite: when it's out, this will leave more time to move forward on SMAPIv3

Lots of work means lots of changes, means: I'm exciting about it. Also sounds more like a 9.0, if that much work is going into it.

@hsnyder AFAIK every - even not so - modern RAID controller can do 'verification read', 'disk scrubbing' or however they call it. It won't fix bitrot with single parity, but it can fix a single and detect dual bit failures.
That's why the only option for our SAN is: RAID6 respectively any DP algorythm.