Guide to getting Fedora CoreOS, Portainer and Xen Orchestra Docker Support working

[foxy82]

I have spent way longer than I should have trying to getting a VM running Fedora CoreOS with Portainer installed as a Docker connected to xscontainer and Xen Orchestra so I thought I'd document the journey.

To be honest once Portainer is installed I'm not sure the rest is really worth the effort to connect Docker to Xen Orchestra it but I wasn't going to let this beat me

If you only want a quick way to get Fedora CoreOS + Portainer up and running steps 5 -> 14 should have you covered.

Also I've seen in the forum that xscontianer isn't supported for Xen 8.2 so is only provided "as-is".

I'm really new to XCP-NG (this VM is only the second I've installed after the Xen Orchestra one so any feedback much appreciated)

1. Download the Fedora CoseOS LiveDVS iso and upload it to XCP-NG ISO Storage(tested with version: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/38.20230430.3.1/x86_64/fedora-coreos-38.20230430.3.1-live.x86_64.iso

On your XCP-NG host

2. Install xscontainer: yum install xscontainer
3. Fix issue with old python2-paramkiko library Note warninig here: https://xcp-ng.org/forum/topic/6845/xscontainer/18)

[Moderator UPDATE 2024-03-19 : DON'T DO THIS. This overrides system libs and "voids the warranty" - Stormi]

yum install python2-pip --enablerepo=epel
pip2 install --upgrade "pip < 21"
pip2 install --upgrade "cryptography == 2.5"
pip2 install --upgrade "paramiko < 3"

4. xscontainer - Apply utf-8 patch (from: https://github.com/xenserver/xscontainer/pull/59/files)

nano /usr/lib/python2.7/site-packages/xscontainer/util/__init__.py

Change

result = str(item) 

to

result = item.encode('utf-8')

On a Linux machine with with Docker or use WSL do the following:

5. Create a password hash for a password to login to Fedore CoreOS:

mkpasswd --method=yescrypt

6. Create a Butane file:

variant: fcos
version: 1.3.0
passwd:
  users:
    - name: core
      password_hash: <YOUR PASSWORD HASH>
      groups:
        - docker
      ssh_authorized_keys:
        - ssh-rsa <YOUR SSH PUBLIC KEY> 
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: |
          docker-vm
    - path: /etc/ssh/sshd_config.d/20-enable-passwords.conf
      mode: 0644
      contents:
        inline: |
          # Fedora CoreOS disables SSH password login by default.
          # Enable it.
          # This file must sort before 40-disable-passwords.conf.
          PasswordAuthentication yes
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          # Tell systemd to not use a pager when printing information
          export SYSTEMD_PAGER=cat
    - path: /etc/sysctl.d/20-silence-audit.conf
      mode: 0644
      contents:
        inline: |
          # Raise console message logging level from DEBUG (7) to WARNING (4)
          # to hide audit messages from the interactive console
          kernel.printk=4
systemd:
  units:
    # Installing software as a layered package with rpm-ostree
    - name: rpm-ostree-install.service
      enabled: true
      contents: |
        [Unit]
        Description=Install software with rpm-ostree
        After=systemd-machine-id-commit.service
        After=network-online.target
        # We run before `zincati.service` to avoid conflicting rpm-ostree transactions.
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # `--allow-inactive` ensures that rpm-ostree does not return an error
        # if the package is already installed. This is useful if the package is
        # added to the root image in a future Fedora CoreOS release as it will
        # prevent the service from failing.
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive xe-guest-utilities-latest
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive nmap
        ExecStart=/bin/touch /var/lib/%N.stamp
        # Now reboot to make changes take effect
        ExecStart=/usr/bin/systemctl reboot
        
        [Install]
        WantedBy=multi-user.target

    # Start software that has been installed
    - name: postinst2.service
      enabled: true
      contents: |
        [Unit]
        Description=Initial System Setup Part 2
        # We run this after the packages have been overlayed
        After=network-online.target
        ConditionPathExists=!/var/lib/%N.stamp
        ConditionPathExists=/var/lib/rpm-ostree-install.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/systemctl enable xe-linux-distribution
        ExecStart=/bin/touch /var/lib/%N.stamp
        # Now reboot to make changes take effect
        ExecStart=/usr/bin/systemctl reboot

        [Install]
        WantedBy=multi-user.target

    - name: docker.portainer.service
      enabled: true
      contents: |-
        [Unit]
        Description=Portainer Admin Container
        After=docker.service
        Requires=docker.service network.target network-online.target

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        TimeoutStartSec=0
        ExecStartPre=-/usr/bin/docker stop %n
        ExecStartPre=-/usr/bin/docker rm %n
        ExecStartPre=/usr/bin/docker pull portainer/portainer-ce
        ExecStart=-/usr/bin/mkdir -p /mnt/shared_nfs/portainer_data
        # Privileged mode is required for binding to local socket to work due to SELINUX (https://github.com/portainer/portainer/issues/849)
        ExecStart=/usr/bin/docker run --privileged=true -d -p 9000:9000 --name %n --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /var/portainer_data:/data portainer/portainer-ce
        ExecStop=/usr/bin/docker stop -t 15 %n

        [Install]
        WantedBy=multi-user.target

7. Convert butane file to ignition file

sudo docker run -i --rm quay.io/coreos/butane:release < coreos-for-xcp-ng.bu > coreos.ign

8. Host the file for the VM

python3 -m http.server

Back on your XC_-NG host:

9. Create a VM.
   Template: CoreOS (probably not needed)
   ISO: Select the Fedora CoreOS ISO
   Disk: Set size to 9GiB (to avoid a warning in the VM)

On the newly created VM console. Host/Port are of the machine hosting the file in step (8):

curl -O <host>:<port>/coreos.ign
sudo coreos-installer install /dev/xvda --ignition-file coreos.ign

Once the install is complete

poweroff

11. On the VM - disable the DVD Drive (in XO this is under the Advanced tab).

12. Restart the VM. It is designed to reboot itself 2 times. -Watch the console for it to finish and don't interrupt it - especially on first power on when it will go to a login prompt for quite a long while - resist the urge to log in and let it finish.

13. Ensure you can login to the VM console with the username: core and password set in step (5)

14. Check Portainer works by going to VM_IP:9000 in a web browser create password and login.

On XCP-NG console:

15. Get the UUID of the VM - either xe vm-list or look in XO.

16. Run

xscontainer-prepare-vm -v <UUID> --username core

select "yes" options throughout.

You should now have CoreOS with Portainer setup and also see the Containers in XO in a "Container" tab when you select the VM.

References:

1. https://discussion.fedoraproject.org/t/fedora-coreos-xentools-installation-for-xenserver-vms-dummy-mode/21337/2
2. https://github.com/xcp-ng/xcp/wiki/Docker-in-XCP-ng
3. https://github.com/xenserver/xscontainer/pull/59
4. https://www.portainer.io/blog/from-zero-to-production-with-fedora-coreos-portainer-and-wordpress-in-7-easy-steps

MattPark created this issue in portainer/portainer

closed SELinux compatibility #849

dalrrard opened this pull request in xenserver/xscontainer

closed Update util/__init__.py #59

dalrrard opened this pull request in xenserver/xscontainer

closed Update util/__init__.py #59

[codycrypto]

I found another workaround (one-step solution) for the "Unable to verify key-based authentication error" without having to mess with any of the python packaging.

This would replace steps 3 and 4 above

Adding

PubkeyAcceptedKeyTypes +ssh-rsa

To your /etc/ssh/sshd_config file will make the VM accept the older authentication

[stormi]

I added a big warning in step 3 which should never be done outside a test host that you are ready to reinstall afterwards.

@codycrypto's comment is another workaround which replaces it without any impact on the host, but weakens the security by accepting weaker key types.