@denis.grilli I understand... but my experience is that even with the default scanning interval the logs become the problem when you get in the range of tens of SRs, thousands of disks. MajorP93's infra is quite small so I believe there is something additional that is spamming the logs... or there is some additional trigger for SR scan.

Update: maybe the default value changed in recent versions?

@Pilow agreed. This shouldn't be the norm. auto-scan-interval=120 is not going to be good for everyone. The majority of people probably don't have any problem with the default value, even in larger deployments.

On the other hand, the real cause of the issue is still elusive.

@MajorP93 Amount of logging is directly proportional to the number of hosts, VMs, SRs and clients (Xen Orchestra, XCP-ng Center...). If you have a lot of those, it's rather normal to have huge logs.

Now, 5 hosts and 2 SRs does not seem to be much so I wouldn't expect you to have problems with huge logs. There could be something going on there. Try restarting your hosts to clear any stuck processes and internal tasks that could potentially spam the logs.

We started having problems with /var/log size when we got in a range of 15+ hosts, 10+ SRs and 1000+ VMs per pool. Unfortunately, log partition cannot be expanded as it is at the end of the disk, followed only by the swap. The workaround we did is to patch the installer to create a large 8GB log partition instead of standard 4GB. Of course, we had to reinstall all of our hosts.

@acebmxer rc in first column means "residual configuration". This means that the package is removed but there are some leftover configuration files so that, for example, when you reinstall the package at later time, the package will use preserved configuration. To remove residual configuration and package completely, use:

sudo apt purge xen-guest-agent

@acebmxer Great. These are some YAML basics. You should read more about it . Following AI instructions without understanding is not going to take you far.

@acebmxer said in Custom config / cloud-init:

...

network:
  version: 2
  ethernets:
    enX0:      # or whatever your interface name is
      dhcp4: false
      addresses: - 10.100.10.206/24
      gateway4: 10.100.10.254
      nameservers:
          addresses:
               - 10.100.10.254
               - 1.1.1.1

Address should be on the next line:

      addresses:
      - 10.100.10.206/24

Regarding 50-cloud-init.yaml, AI is lying .

@nuentes metadata in this context is just XAPI database. In other words, it only contains information about your VMs, SRs, networks, pools etc. It does not contain anything system level. It is not a backup of the host system.

As far as I know, but someone from Vates can confirm, metadata backup functionality in XO is based on XAPI pool-dump-database command:

xe pool-dump-database file-name=dump.xml

There is some info about it here:

https://docs.xenserver.com/en-us/xenserver/8/dr/backup.html

P.S. I guess metadata backup is also XML just like XAPI state file (database). I don't know why JSON came to my mind regarding metadata backup.

@Greg_E Theoretically yes. I've never been in such situation so I would have to learn, experiment and improvise along the way. If you can mount the file system of the host, you can find XAPI database in this location:

/var/lib/xcp/state.db

This is an XML formated file. I don't know if this is the same format as metadata backup (I think it's JSON instead). So you could possibly restore this file to proper location and restart the host but you would probably need to change some references in it, like UUIDs of local file systems on a freshly installed system. Other possibility is that the state file could be converted to metadata backup and imported but I'm not aware of there being any such conversion tool. It would have to be improvised.

There is always alot you can do. It just depends how deep you want to go, how complex you can go and how much time you can spare.

@acebmxer Did you do:

cloud-init clean --logs --seed

before converting the VM to template?

Also, network configuration is not part of the cloud-config (aka user data). In XO, there is a separate field called "Network config" where it should be specified. See examples at the end of the guide I pasted earlier. network: key should also be removed (commented in the examples).

@nuentes Oh no, no. Your system is not destroyed beyond repair. It can be repaired. It's just that it is almost impossible or too much of a hustle for anyone to try to help you over forum. Someone has to sit in front of your machine to do it.

My only guess is that ChatGPT instructed you to make changes based on a CentOS system but XCP-ng and Xen virtualization in general is much different than regular CentOS. It has two stage boot process. First the Xen kernel boots and then a special virtual machine called Dom0 is booted. What you are accessing and reconfiguring is in fact this VM, not the underlying "system". So it's like a two layer system and some configuration must be done on Xen layer, some on Dom0 layer. I'm unfortunately unfamiliar with exact specifics on kernel and initrd image generation for this case so I can't spot where thing have gone wrong.

In short terms. Instead of going back and forth and trying a lot of different things, it's more time saving and simpler to reinstall the system and restore metadata if you already have a backup.

@nuentes No. My intention was to rise awareness of USB (un)reliability, especially the reliability of USB attached storage. Also, either I'm blind or there is no mention of your system not being installed on USB storage.

You said everything yourself. Your problems started with USB which you assumed can be fixed by flicking some kernel parameters. In the process of "fixing", you destroyed your system. Unfortunately, I believe that the system is now beyond repair via interactive forum session because no one knows what really happened. Backup is your best friend.

@Pilow Ah, yes. Makes much more sense now. My mind was too focused on Vates stack that I didn't think about anything outside of it like DNS, 1password, firewall etc. integration.

You seem to have a quite good vision of what you are going to sell, both on technical level and business level. I work for cloud/managed services provider myself and we grew large but never got to this level of integration. I'm envious now

@Pilow Long live the XenCenter/XCP-ng Center king

@Pilow said in Racked today, entire hosting solution based on Vates stack:

... and the fact that Pulumi can go beyond XO provider to manage other aspects of the stack

Can you elaborate more?

@nuentes said in Booting to Dracut (I trusted ChatGPT):

Other things that I didn't mention yet that may or may not be relevent:

• the enclosure is connected with a USB-C cable to a USB-C input
• the enclosure hosts 4 disks
• Other disks/enclosures were not experiencing the disconnect issue
• The enclosure was actually working fine until a few days after I troubleshot/resolved an issue with one disk having slow transfer speeds. I switched the enclosure from USB-A to USB-C and also disabled spindown for the affected disk in the enclosure. Disabling spindown was done at the VM level, so I didn't mention it above.

there is your problem

USB is very unreliable for any kind of serious data transfer. Disconnecting devices, data transfer errors, corruptions are just the tip of the iceberg. For anything reliable you have to go for network attached storage or eSATA.

I'm that type of a person that verifies md5 sums of all files copied to the USB flash, disk, enclosure etc. I've spotted data corruptions sooooo many times regardless of the OS, version, HW, USB type, storage device type... The only common thing was USB. eSATA and ethernet never produced such corruptions. As far as I'm concerned, USB is for mice and keyboards... and maybe a lamp or fan

@Pilow just curious. Why use Pulumi for VM provisioning instead of XO API directly? You are interfacing with XO API for other stuff anyway, right?

@Pilow password: as a global option and passwd: or plain_text_passwd: under users: key are two different things. The first one sets the password for the default user, ubuntu on Ubuntu if I recall correctly, while the others set password for the user specified in the users: key.

Read the docs people

@acebmxer said in Custom config / cloud-init:

If convert this newly created vm to a template. Then create another vm from this template. Even though the VM is generating a new mac address when the vm boots its still getting / using the same ip from vm used to create the template.

You will have to carefully read cloud-init documentation to understand how it works and what phases it has.

Short answer is that VM template creation is a bit more work than just preparing some base VM and than converting it to the template or use it to clone other VMs. For a start, cloud-init has to be "cleaned" so that it is reset to the initial state.

cloud-init, or better said most of it's modules, run only once - just on first boot. So... when you create a VM from a hub template, cloud-init will run but if you use that VM to clone another VM, cloud-init will not run like on first boot of the original VM. Aside from that, to prepare some VM to be a template for other VMs, other "stuff" beside cloud-init has to be cleaned up like machine-id, SSH host keys, network configuration, logs... There is some official documentation on this here:

https://docs.xcp-ng.org/guides/create-use-custom-xcpng-ubuntu-templates/

and some forum addendum here:

https://xcp-ng.org/forum/topic/11008/ubuntu-22.04.5-custom-template-additional-steps-missing-from-documentation

read carefully

@acebmxer said in Custom config / cloud-init:

Think I figured it out. first i was using the wrong key. A lot of going back and forth trying new keys forgot to swap back to original...

new config...

#cloud-config
hostname: {name}
users:
  - name: newusername
    gecos: New User
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3....18ZbA

Any suggestions should I not use something in the above config if the server was in production?

• Well... NOPASSWD:ALL can be considered a security issue because user is not required to type a password to gain root privileges. If someone gains access to this user via for example stolen SSH key or some exploit, it will automatically have access to the root user.

• It is enough to put the user into "sudo" group for it to gain sudo privileges (with password required) because there is already this global sudoers rule:

  %sudo ALL=(ALL:ALL) ALL

• Adding the user to users and admin groups is a little bit contradictory. It should either be users or admin. Also, admin group does not exist on Ubuntu 24.04. It existed in some earlier versions but I'm not sure when and why it disappeared. It's best to just add the user to the sudo group so it can run commands as root and adm group (yes adm, not "admin") so it can view system logs (/var/log) without using sudo. This is if you are creating admin type user. If the user is just a regular user, you can add it to the users group but it's not necessary. If you are confused, don't worry, I'm too .

This is for Ubuntu, other operating systems have other principles and rules so there is no universal solution.

@Pilow I'm not so impressed with the hardware stack ... but this Cloudbox GUI is much more interesting. Is this something you developed from scratch in house for VMware and that ported to XO or...?