Skip to content. | Skip to navigation

Navigation

You are here: Home / Support / Guides / Tools / Kubernetes / Setup

Personal tools

 

Setup

VMs

In this example, we're using the previously mentioned

  • one master node: 2 vCPU 4GB RAM 40GB disk
  • two worker nodes: 4 vCPU 16GB RAM 40GB disk

(total overkill!)

This example is running Fedora 35 on those VMs.

Per-VM Setup

Broadly following https://kubernetes.io/docs/setup/ ...

You only need to do the SWAP, containerd and kubernetes sections once per node but on all nodes.

SWAP

It must be gone. The installer will cat /proc/swaps and if anything is mentioned it will fail.

For Fedora:

# dnf remove -y zram-generator-defaults

(you may need to reboot, maybe run swapoff -a, maybe get generally annoyed)

The acid test is:

# cat /proc/swaps
Filename                                Type            Size            Used            Priority

We're good.

containerd

Following https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

containerd requires a couple of kernel modules:

# cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

# sudo modprobe overlay
# sudo modprobe br_netfilter

# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# sudo sysctl --system

before we can install it and (re-)generate the default config:

# dnf install -y containerd
# containerd config default | sudo tee /etc/containerd/config.toml

We need to edit that default config to assert the use of SystemD's cgroup mechanism. There's a long discussion about cgroups(7) which you can internalize before just going with the flow:

# vi +/runc.options /etc/containerd/config.toml
            SystemdCgroup = true

# systemctl enable --now containerd

Kubernetes

Following https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

The main control interface, if you like, will be listening on port 6443. You may want to open that up through, say, OpenStack, or any other intervening systems if you want to be able to manipulate Kubernetes from your desktop.

Technically, Kubernetes requires some kernel modules as well but these happen to overlap with containerd, above. For completeness:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

Disable SELinux:

# sudo setenforce 0
# sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

For Fedora, we need to install a new repo, noting that it uses el7 binaries (shouldn't make any odds...):

# cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

Notice we exclude the three main binaries in the above repo then enable then just for the following installation line:

# sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

# sudo systemctl enable --now kubelet

which means that the three binaries will remain in lockstep.

Master Installation

Options

There are quite a few options here for which we can get a view of what might happen with:

# kubeadm config print init-defaults

and kubeadm init -h prints out the flags which let us manipulate the defaults.

  • it seems common enough to set the Pod network CIDR with --pod-network-cidr=10.254.0.0/16 -- and see the warning in Calico, below

  • the Service CIDR defaults to --service-cidr=10.96.0.0/12

  • the Service DNS domain defaults to --service-dns-domain cluster.local

    .local is, of course, in breach of RFC 6762 Section 3 although this is far from the first such breach, see https://en.wikipedia.org/wiki/.local

  • although the localAPIEndPoint.advertiseAddress is listed as 1.2.3.4 it appears to be 0.0.0.0 in practice

  • most usefully, we can add some subjectAlternateNames to the generated certificate

    This is particularly useful to us, with our OpenStack hats on, as the node's name and IP are internal and not necessarily going to match what external users will use.

    The format is --apiserver-cert-extra-sans "<IP> <host> ..." (noting that this is not quite the same as the IP:<IP> DNS:<host> format that the certificate itself uses).

    Again, OpenStack might prompt us to use constructions like $(curl -s http://169.254.169.254/2009-04-04/meta-data/public-ipv4) to get the VM's assigned Floating IP address.

    Note, though, that these are extra SANs and kubeadm does generate some default values including various hostnames and IP addresses.

Exceptions

Not everything gets a flag, though. Notably, the cluster is going to get a default name, kubernetes, which is fine on its own, unless you want a different one for which there are no available option flags.

Here, we would have to pass a YAML configuration file, --config FILE, with the appropriate option:

clusterName: soho-cluster

Install

Very simply:

# kubeadm init <options>

So, maybe:

# kubeadm init --pod-network-cidr=10.254.0.0/16 --apiserver-cert-extra-sans "$(curl -s http://169.254.169.254/2009-04-04/meta-data/public-ipv4)"

Important

kubeadm init will print out the appropriate kubeadm join command for the worker nodes which includes a token and a certificate hash. Make a note of the token as it has a 24hr expiry date.

You can create more tokens with kubeadm token create and recover the hash with:

# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
  openssl dgst -sha256 -hex | sed 's/^.* //'
admin.conf

kubeadm init will have created /etc/kubernetes/admin.conf which you, as the Kubernetes administrator, might want to copy to wield full power over the cluster.

Nominally, it might be copied to ~/.kube/config although equally, so long as it appears in the KUBECONFIG environment variable kubectl will be able to use it.

In the rest of this example, I'll use # as the prompt for kubectl using admin.conf (as ~/.kube/config or whatever) and $ as the prompt for kubectl with a less permissive configuration.

Calico

Calico is easy to install but a quick word of warning.

Warning

If you didn't define the Pod CIDR with kubeadm init, above, then Calico will use 192.168.0.0/16 and cannot be undone. Which is likely to conflict with your site networking and therefore be annoying.

Download the Calico YAML file and then apply it:

# curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O
# kubectl apply -f calico.yaml

If you're quick (OK, within 10s of seconds!) you can watch Calico bursting into life:

# kubectl get all -n kube-system

looking for the Pods to become ready.

Although ultimately you want to see that CoreDNS is running:

# kubectl cluster-info
Kubernetes control plane is running at https://<node-IP>:6443
CoreDNS is running at https://<node-IP>:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Worker Installation

Having done the one-off per-node steps you can simply run the kubeadm join command printed out earlier:

# kubeadm join <master-node-IP>:6443 --token TOKEN \
       --discovery-token-ca-cert-hash sha256:HASH

Node Removal

On the master node, say:

# kubectl get nodes
# kubectl drain <node-name> --delete-local-data --force --ignore-daemonsets
# kubectl delete node <node-name>

and then on <node-name> itself:

# kubeadm reset

Although the suggestion is that there'll be further work to tidy up iptables, /etc/* and dnf.

Document Actions