Installing Fedora CoreOS

In this article, we’re going to install Fedora CoreOS using an Ignition config file. We’re going to discuss the Butane transpiler and common settings and pitfalls.

Welcome to the stage: Fedora CoreOS

The first step for your K8s deployment is going to be to pick the Linux distribution that you want to use for the system. You can go with something tried and tested like Ubuntu, RHEL or CentOS. The disadvantage of such distributions is that they can carry additional bloat, that you don’t need for your kubernetes system to work.

In recent years, several distributions purpose-built for container workloads have emerged, including Fedora CoreOS (backed by Red Hat), Flatcar Container Linux, Rancher OS (backed by SUSE), and AWS’ Bottlerocket. In this project, we’re going to go with CoreOS, just because I have had great success with the OS. I think Flatcar works well, but I haven’t personally used it. Bottlerocket only runs on EC2 (not on-premise).

One of the most interesting features of CoreOS is how you set it up and keep it updated:

• Set-up: CoreOS uses Ignition as its provisioning system. Ignition operates during the first boot and allows you to configure the system declaratively. This includes setting up disks, filesystems, networking, and injecting SSH keys. Since Ignition runs before the OS is fully booted, it ensures a clean and reproducible setup.

  This allows you to treat your systems as cattle and not pets. Don’t SSH into the machine and patch it yourself. Instead, if anything breaks, just replace the machine (e.g, create a new VM, or reinstall the OS on bare-metal). Because we’re using Infrastructure as Code (more on that later, and also in the GitOps article), you can be sure that — given the automation — you’ll have a fully functioning node at the end of the process.

• Updates: Zincati is the update agent for Fedora CoreOS. It integrates with rpm-ostree to manage updates and ensures that updates are applied safely. In order to do that, it employs phased rollouts and automatic reboots when needed. This makes updates seamless and ensures the system stays secure and up-to-date without manual intervention.

In other words: You set up the system once, using your Ignition config. Then, the cluster runs itself and nodes automatically update, and coordinate among themselves to do so safely.

Setting up CoreOS with Ignition and Butane

In CoreOS, Ignition is the system responsible for applying configurations during the first boot of a CoreOS instance. It operates early in the boot process, ensuring that the system is set up exactly as specified before the operating system is fully initialized. Ignition files are just JSON, but tend to be a bit tedious to write by had. That’s why there’s Butane, a transpiler from a YAML file to an Ignition config. Butane is a human-friendly utility that converts YAML configuration files into the JSON format required by Ignition. It allows users to define complex system configurations, such as files, users, networking, and disk partitions, in a simplified and readable manner.

Creating a user that can SSH into the machine

The first step in our butane config is creating a user that ansible will use to later SSH into the machine:

variant: fcos
version: 1.6.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa YOURSSHKEYHERE

The code should be relatively self-explanatory. Alternatively, instead of an SSH key, you could also set a password here.

Partitioning the disks

Next, you can partition the disks like this:

storage:
  disks:
    - # The link to the block device the OS was booted from.
      device: /dev/sda
      # We do not want to wipe the partition table since this is the primary
      # device.
      wipe_table: false
      partitions:
        - number: 4
          label: root
          # Allocate at least 8 GiB to the rootfs. See NOTE above about this.
          size_mib: 0
          resize: true
  filesystems:
    - device: /dev/disk/by-partlabel/root
      wipe_filesystem: true
      format: xfs
      label: root

This is a sane default, that uses /dev/sda. Partition number 4 is the root partition (the others are for boot and the like). CoreOS requires this partition to be at least 8 GiB. The configuration file above sets it to autoexpand to fill the entire drive. It also formats the rootfs to use the XFS filesystem. Alternatively, you could choose btrfs or ext4.

My config doesn’t have a swap partition. If you have any other needs, such as RAID, more/ different partitions, or LUKS encryption do check out the official CoreOS docs.

Networking

If you leave networking uninitialized in your Ignition file, by default CoreOS will use DHCP for all available interfaces. Alternatively, you can also configure network manager using the file drop-in:

storage:
  files:
    - path: /etc/NetworkManager/system-connections/${interface}.nmconnection
      mode: 0600
      contents:
        inline: |
          [connection]
          id=${interface}
          type=ethernet
          interface-name=${interface}
          [ipv4]
          address1=${ip}/${prefix},${gateway}
          dhcp-hostname=${hostname}
          dns=${nameserver};
          dns-search=
          may-fail=false
          method=manual

Similarly, you can set the hostname like this:

storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: example.fschoenberger.dev

Set the keymap

If you happen to not have an american keyboard, here’s how you’d change the keymap:

storage:
  files:
    - path: /etc/vconsole.conf
      mode: 0644
      contents:
        inline: KEYMAP=de

Using FleetLock for update coordination

As hinted at earlier, CoreOS uses Zincati to install updates. For some updates, a reboot might be required, which makes part of our cluster unavailable. Hence, to minimize disruptions, we would like to coordinate reboots, such that not large parts of the cluster go offline at the same time. The way Zincati achieves this, is using the FleetlLock Protocol. We’re going to run a coordination server in our cluster that allows only one node at a time to be offline.

In one of the following articles, we’re going to deploy FleetLock, but for now, we need to add the following configuration:

storage:
  files:
    - path: /etc/zincati/config.d/55-update-strategy.toml
      contents:
        inline: |
          [updates]
          strategy = "fleet_lock"
          [updates.fleet_lock]

          # CHANGE THIS TO FIT INTO YOUR NETWORK
          base_url = "http://10.3.0.15/"

Transpiling the file

After you’re done with your config, you can finally compile it with

$ docker run --interactive --rm --security-opt label=disable \
    --volume ${PWD}:/pwd \
    --workdir /pwd \
    quay.io/coreos/butane:release \
    --pretty \
    --strict \
    your_config.bu > transpiled_config.ign

Installing the OS

Now is the time to install the operating system. If you’re lucky enough to have your own DHCP server, this would be an excellent time to use PXE boot. For everyone else, it’s also fine to just flash the ISO to a USB drive (e.g., using Rufus), and boot from that.

Then, you can run coreos-installer install (docs) to install your system, for example like this:

$ coreos-installer install --ignition-url <URL> /dev/sda

If you can control your DHCP leases to a satisfying degree, you could also be interested in the --copy-network switch.

So that’s it! You’ve successfully installed Fedora CoreOS. Try SSHing into the machines. If you find bugs in your ignition files, don’t change the machines and instead just reinstall with the new ignition file.

In the next article, we’re going to use Ansible and Kubespray to deploy kubernetes cluster!