Posts made by nikade

@fred974 said in How to pass MAC address to pfSense VM:

@nikade do you use 2 IP subnet? 1x for xcp-ng management + ssh and another extra that you use for the VMs or is it all 1 single subnet? Also did you requested separate MAC on Hetzner backend or left it as is?

@nikade said in How to pass MAC address to pfSense VM:

We then enabled forwarding on the xcp-ng host and setup the first usable IP as "local" subnet

How did you achieve this? Thank you very much in advance. I really appreciate the feedback..

1 IPv4 on the dom0, the one that came with the server. Then we ordered a /29 IPv4 subnet which is routed to the dom0 IPv4 address.
You then need to setup IPv4 forwarding on the dom0 by editing the sysctl and then setup the 1st usable IPv4 from the /29 IPv4 subnet on an internal interface.

I just created a dummy vlan interface and put it there and that worked fine.

Used Hetzner many times with xcp-ng and pfsense, we ordered a subnet which was routed to the xcp-ng host.
We then enabled forwarding on the xcp-ng host and setup the first usable IP as "local" subnet and then we were able to use the rest of the IP's on our VM's using that first IP as default gateway.

@planedrop said in Backup / Migration Performance:

@KPS Regarding the 2TiB limitation, it'll definitely be nice when we have SMAPIv3 so we can go over this, but it's worth noting that IMO no VMs should be larger than this anyway. Generally speaking if you need that kind of space it'd be better to just use a NAS/iSCSI setup. Something like TrueNAS can delivery that at high speed, and then handle it's own backups and replication of it.

I know most probably already know this, and all environments are different (I manage one that requires a 7TiB local disk, at least for the time being, plan is to migrate it to a NAS once the software vendor supports it), but it's worth noting anytime I see the 2TiB limit come up, ideally it should be architected around so the VMs are nimble.

I do something similar w/ a pretty massive SMB share and TrueNAS can back this up at whatever speed the WAN can handle, in my case 2 gigabits and it'll maintain that 2 gigabit upload for 8+ hours without slowing down. (and I'm confident even 10 gigabit would be possible with this box)

We have 1 exception and that is for Windows file servers which is backing our DFS.
Except from those we dont allow VM's larger than 1Tb and if they're that big we do not back them up because it usually breaks and cause all kinds of problems.

@Greg_E yeah I can imagine, I dont think the ram does much except gives you some "margins" under some heavier load. With 12 VM's I wouldn't bother increasing it from the default values

Yea the mgmt interface in ESXi is nice, I think the standalone interface is almost better than vCenter. Im pretty sure XO Lite will be able to do more when it is done, for example you'll be able to manage a small pool with it.

Googling the error "supervisor read data page not present" gives a lot of hints towards bad memory, are you using ECC or non ECC?

@Greg_E said in What should i expect from VM migration performance from Xen-ng ?:

@nikade

I'm doing upgrades on both XCP-NG and Truenas, when they are done I have 4 small VMs that I'm going to fire up at the same time and see what I can see. I might be able to hit 2gbps reads while all 4 are booting.

I have one last thing to try in my speed testing, going to see if increasing the RAM for each host makes a difference. Probably not since each already has a decent amount (defaults go up when you pass a certain amount of available). I read somewhere that going up to 16GB for each host might help, with my lab being so lightly used, I might try going up to 32GB for fun. I have lots of slots in my production system where I can add RAM if this helps on lab, the lab machines are full.

We usually set our busy hosts to 16Gb for the dom0 - It does make a stability difference in our case (We could have 30-40 running VM's per host) especially when there is a bit of load inside the VM's.
Normal hosts gets 4-8Gb ram depending on the total amount of ram in the host, 4Gb on the ones with 32Gb and then 6Gb for 64Gb and 8Gb and upwards for the ones with +128Gb.

@nikade they have always been created using the "other" template.
The original VM's had 1 vCPU and 2 GB preset on the template. Aside from the values changed initially, I didn't made changes to the memory/resources.
NIC: 2 NIC's (one for WAN another for LAN). Initially they always have the "Realtek" driver, so all I did on one test was changing that to Intel e1000. But there was no change to the outcome.

Hi,

Can you go to the "Advanced" tab of the VM and show the memory settings?
I want to make sure you're not using dynamic memory since it is really unstable in Xen.

What happends if you create a new VM with the template "other" and attempt to make a clean installation? Do you get the same error?

Also, what kind of configuration have to given the machine? For example NIC? RAM? Dynamic RAM or static?

@Greg_E said in What should i expect from VM migration performance from Xen-ng ?:

@nikade

I've spent a bunch of time trying to find some dark magic to making the VDI migration faster, so far nothing. My VM (memory) migration is fast enough that I'm not concerned right now. and don't have any testing to show for it.

Currently migrating the test VDI from storage1 to storage2 (again) and getting an average of 400/400mbps (lower case m and b). If I do three VDI at once, I can get over a gigbit and sometimes close to 2 gigabit.

It's either SMAPIv1 or it is a file "block" size issue, bigger blocks can get me benchmarks up to 600MBps to almost 700MBps (capital M and B) on my slow storage over a 10gbps network. Testing this with XCP-NG 8.3 release to see if anything changed from the Beta, so far all is the same. Also all testing done with thin provisioned file shares (SMB and NFS). If I could get half my maximum tests for the VDI migration, I'd be happy. In fact I'm extremely pleased that my storage can go as fast as it is showing, it's all old stuff on SATA.

I have a whole thread on this testing if you want to read more.

You can see the migrate which was 400/400 and then the benchmark across the ethernet interface of my Truenas, this example was migrate from SMB to NFS, and benchmark on the NFS. Settings for that NFS are in the thread mentioned and certainly my fastest non-real world performance to date.

That's impressive!
We're not seeing as high speeds are you are, we have 3 different storages, mostly doing NFS tho. We're still running 8.2.0 but I dont really think it matters as the issue is most likely tied to the SMAPIv1.

We also noted that it goes a bit faster when doing 3-4 VDI's in parallell, but the individual speed per migration is about the same.

@tuxpowered said in Rolling Pool Update - host took too long to restart:

@nikade Both clusters have shared storage. Kind of a pre-req to have a cluster
Yes both systems were on line and working well. One is TrueNas SCALE and the other is a Qnap. Both massively over kill systems.

Yea I was hoping that you were using shared storage, but i've actually seen ppl using clusters/pools without shared storage so I felt I had to ask

@henri9813 said in What should i expect from VM migration performance from Xen-ng ?:

Hello,

Thanks for your answer @nikade

I migrate both storage and vm.

I have 25Gb/s nic, but I have a 5Gb/s limitation at switch level.

But I found on another topic an explanation about this, this could be related to SMAPIv1.

Best regards

Yeah, when migrating storage the speeds are pretty much the same no matter what NIC you have...
This is a known limitation and I hope it will be resolved soon.

@tuxpowered justs to make sure, you're using the "Rolling pool update" button, right?
And then the master is patched, vm's migrated off and then no bueno, correct?

Did you happend to have any shared storage in this pool or is all storage local storage?

@john-c said in Please review - XCP-ng Reference Architecture:

@nikade said in Please review - XCP-ng Reference Architecture:

@john-c said in Please review - XCP-ng Reference Architecture:

TrueSecure

Whats that? Never heard of TrueSecure on TrueNAS.

It's an application or feature for TrueNAS Scale, TrueNAS Core and/or TrueNAS Enterprise. Which enables the enabling and configuration of security features of TrueNAS instances (software and/or hardware).

https://www.truenas.com/blog/truenas-security-in-2024/

Alright - I didnt know that, thanks for the info.

@john-c said in Please review - XCP-ng Reference Architecture:

TrueSecure

Whats that? Never heard of TrueSecure on TrueNAS.

@TS79 I dont think it really matters, we run ours in one of our pools and we've been doing that since 2016 without any issues.

@andrewperry yeah try set it 16/16Gb instead, it will probably do some magic

@henri9813 said in What should i expect from VM migration performance from Xen-ng ?:

Hello,

So, does someone know why I can't go over 145MB/s and how to increase the vdi speed transfert ?

( I have 5Gb/s between nodes )

Thanks !

Are you migrating storage or just the VM?
How can you have 5Gb/s between the nodes? What kind of networking setup are you using?

And you guys aren't using any kind of dynamic memory?
Can you post a screen dump of the Advanced tab where it shows the memory configuration?

We have VM's with 128Gb ram that migrates just fine, when migrating it between hosts the network shows peaks at 7,6Gbit/s and it is migrated in about ~20 seconds.
Smaller VM's with 8, 16 or even 32Gb ram is migrated almost instantly.

@flakpyro said in Switching to XCP-NG, want to hear your problems:

@CodeMercenary I am using V4 on the XO-Server to our backup remotes and it seems to work just fine. However using V4 as a storage SR was nothing but problems, as @nikade mentioned we had tons of NFS Server not responding issues which would lock up hosts and VMs causing downtime. Since moving to v3 that hasn't happened.

Checking a host's NFS retransmissions stats after 9 days of uptime i see we have had some retransmissions but they have not caused any downtime or even any timeout messages to appear in dmesg on the host.

[xcpng-prd-02 ~]# nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
268513028   169        268537542

From what a gather from this blog post from redhat (https://www.redhat.com/sysadmin/using-nfsstat-nfsiostat) it seems like that amount of retransmissions is VERY low and not an issue.

Thats fine, we've got a lot more and I haven't seen any "nfs server not responding" in dmesg yet.
Using NFS v3 for both SR and backups now for a couple of years and it's been great, I think I had issues once or twice in like 5-6 years on the backup SR where the vhd file got locked by dom0, Vates helped out there as always and it was resolved quickly.

@flakpyro said in Switching to XCP-NG, want to hear your problems:

@rfx77 As a follow up v4 did cause more downtime for us, i switched over everything to v3 which has been much better so far. Going to be curious to see how a controller failover goes during a firmware update as v3 is stateless vs v4 and iscsi being stateful protocols.

We're also using v3 and failover on our Dell Powerstore's are seamless.
Haven't tried v4 since we had A LOT of "nfs server not responding" issues with it and immediately went back to v3.