Guide to getting Fedora CoreOS, Portainer and Xen Orchestra Docker Support working

foxy82

I have spent way longer than I should have trying to getting a VM running Fedora CoreOS with Portainer installed as a Docker connected to xscontainer and Xen Orchestra so I thought I'd document the journey.

To be honest once Portainer is installed I'm not sure the rest is really worth the effort to connect Docker to Xen Orchestra it but I wasn't going to let this beat me

If you only want a quick way to get Fedora CoreOS + Portainer up and running steps 5 -> 14 should have you covered.

Also I've seen in the forum that xscontianer isn't supported for Xen 8.2 so is only provided "as-is".

I'm really new to XCP-NG (this VM is only the second I've installed after the Xen Orchestra one so any feedback much appreciated)

Download the Fedora CoseOS LiveDVS iso and upload it to XCP-NG ISO Storage(tested with version: https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/38.20230430.3.1/x86_64/fedora-coreos-38.20230430.3.1-live.x86_64.iso

On your XCP-NG host

Install xscontainer: yum install xscontainer
Fix issue with old python2-paramkiko library Note warninig here: https://xcp-ng.org/forum/topic/6845/xscontainer/18)

[Moderator UPDATE 2024-03-19 : DON'T DO THIS. This overrides system libs and "voids the warranty" - Stormi]

yum install python2-pip --enablerepo=epel
pip2 install --upgrade "pip < 21"
pip2 install --upgrade "cryptography == 2.5"
pip2 install --upgrade "paramiko < 3"

xscontainer - Apply utf-8 patch (from: https://github.com/xenserver/xscontainer/pull/59/files)

nano /usr/lib/python2.7/site-packages/xscontainer/util/__init__.py

Change

result = str(item)

to

result = item.encode('utf-8')

On a Linux machine with with Docker or use WSL do the following:

Create a password hash for a password to login to Fedore CoreOS:

mkpasswd --method=yescrypt

Create a Butane file:

variant: fcos
version: 1.3.0
passwd:
  users:
    - name: core
      password_hash: <YOUR PASSWORD HASH>
      groups:
        - docker
      ssh_authorized_keys:
        - ssh-rsa <YOUR SSH PUBLIC KEY> 
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: |
          docker-vm
    - path: /etc/ssh/sshd_config.d/20-enable-passwords.conf
      mode: 0644
      contents:
        inline: |
          # Fedora CoreOS disables SSH password login by default.
          # Enable it.
          # This file must sort before 40-disable-passwords.conf.
          PasswordAuthentication yes
    - path: /etc/profile.d/systemd-pager.sh
      mode: 0644
      contents:
        inline: |
          # Tell systemd to not use a pager when printing information
          export SYSTEMD_PAGER=cat
    - path: /etc/sysctl.d/20-silence-audit.conf
      mode: 0644
      contents:
        inline: |
          # Raise console message logging level from DEBUG (7) to WARNING (4)
          # to hide audit messages from the interactive console
          kernel.printk=4
systemd:
  units:
    # Installing software as a layered package with rpm-ostree
    - name: rpm-ostree-install.service
      enabled: true
      contents: |
        [Unit]
        Description=Install software with rpm-ostree
        After=systemd-machine-id-commit.service
        After=network-online.target
        # We run before `zincati.service` to avoid conflicting rpm-ostree transactions.
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # `--allow-inactive` ensures that rpm-ostree does not return an error
        # if the package is already installed. This is useful if the package is
        # added to the root image in a future Fedora CoreOS release as it will
        # prevent the service from failing.
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive xe-guest-utilities-latest
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive nmap
        ExecStart=/bin/touch /var/lib/%N.stamp
        # Now reboot to make changes take effect
        ExecStart=/usr/bin/systemctl reboot
        
        [Install]
        WantedBy=multi-user.target

    # Start software that has been installed
    - name: postinst2.service
      enabled: true
      contents: |
        [Unit]
        Description=Initial System Setup Part 2
        # We run this after the packages have been overlayed
        After=network-online.target
        ConditionPathExists=!/var/lib/%N.stamp
        ConditionPathExists=/var/lib/rpm-ostree-install.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/bin/systemctl enable xe-linux-distribution
        ExecStart=/bin/touch /var/lib/%N.stamp
        # Now reboot to make changes take effect
        ExecStart=/usr/bin/systemctl reboot

        [Install]
        WantedBy=multi-user.target

    - name: docker.portainer.service
      enabled: true
      contents: |-
        [Unit]
        Description=Portainer Admin Container
        After=docker.service
        Requires=docker.service network.target network-online.target

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        TimeoutStartSec=0
        ExecStartPre=-/usr/bin/docker stop %n
        ExecStartPre=-/usr/bin/docker rm %n
        ExecStartPre=/usr/bin/docker pull portainer/portainer-ce
        ExecStart=-/usr/bin/mkdir -p /mnt/shared_nfs/portainer_data
        # Privileged mode is required for binding to local socket to work due to SELINUX (https://github.com/portainer/portainer/issues/849)
        ExecStart=/usr/bin/docker run --privileged=true -d -p 9000:9000 --name %n --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /var/portainer_data:/data portainer/portainer-ce
        ExecStop=/usr/bin/docker stop -t 15 %n

        [Install]
        WantedBy=multi-user.target

Convert butane file to ignition file

sudo docker run -i --rm quay.io/coreos/butane:release < coreos-for-xcp-ng.bu > coreos.ign

Host the file for the VM

python3 -m http.server

Back on your XC_-NG host:

Create a VM.
Template: CoreOS (probably not needed)
ISO: Select the Fedora CoreOS ISO
Disk: Set size to 9GiB (to avoid a warning in the VM)

On the newly created VM console. Host/Port are of the machine hosting the file in step (8):

curl -O <host>:<port>/coreos.ign
sudo coreos-installer install /dev/xvda --ignition-file coreos.ign

Once the install is complete

poweroff

On the VM - disable the DVD Drive (in XO this is under the Advanced tab).
Restart the VM. It is designed to reboot itself 2 times. -Watch the console for it to finish and don't interrupt it - especially on first power on when it will go to a login prompt for quite a long while - resist the urge to log in and let it finish.
Ensure you can login to the VM console with the username: core and password set in step (5)
Check Portainer works by going to VM_IP:9000 in a web browser create password and login.

On XCP-NG console:

Get the UUID of the VM - either xe vm-list or look in XO.
Run

xscontainer-prepare-vm -v <UUID> --username core

select "yes" options throughout.

You should now have CoreOS with Portainer setup and also see the Containers in XO in a "Container" tab when you select the VM.

References:

MattPark created this issue in portainer/portainer

closed SELinux compatibility #849

dalrrard opened this pull request in xenserver/xscontainer

closed Update util/init.py #59

dalrrard opened this pull request in xenserver/xscontainer

closed Update util/init.py #59

codycrypto

I found another workaround (one-step solution) for the "Unable to verify key-based authentication error" without having to mess with any of the python packaging.

This would replace steps 3 and 4 above

Adding

PubkeyAcceptedKeyTypes +ssh-rsa

To your /etc/ssh/sshd_config file will make the VM accept the older authentication

stormi

I added a big warning in step 3 which should never be done outside a test host that you are ready to reinstall afterwards.

codycrypto's comment is another workaround which replaces it without any impact on the host, but weakens the security by accepting weaker key types.

Guide to getting Fedora CoreOS, Portainer and Xen Orchestra Docker Support working

closed SELinux compatibility #849

closed Update util/__init__.py #59

closed Update util/__init__.py #59

closed Update util/init.py #59

closed Update util/init.py #59