Build a Two-Node kubeadm Cluster on GCP

Part 1 of the CKA roadmap · Phase 0 — a cluster of your own

Every troubleshooting task on the exam assumes you know what a healthy cluster looks like. The fastest way to learn that is to build one by hand, break it, and put it back — which is why this comes before anything else in the roadmap.

A browser sandbox or minikube gets you a working cluster in a minute, and hides exactly the parts worth learning: which machine you are on, what kubeadm writes to disk, and why a pod network can look fine while no packet crosses between nodes. I broke this cluster twice in two days. Both breakages are below, because they taught me more than the build did.

The task

Two fresh Ubuntu 24.04 VMs, cp and worker. Bring up Kubernetes 1.33 with kubeadm and a pod network that works across both nodes. Prove it: a pod running on worker must reach a pod IP on cp.

What you are building


Nodes	`cp` (control plane) and `worker`
OS	Ubuntu 24.04 LTS, x86-64
Machine type	`e2-medium` — 2 vCPU, 4 GB. kubeadm refuses a control plane with fewer than 2 CPUs
Kubernetes	1.33, to match the exam
CNI	Flannel — one manifest, no operator
Pod CIDR	`10.244.0.0/16`
Region	`asia-southeast1`, though any works

Flannel is the simplest CNI to install, which is why it is the default here. It has one serious limitation — it does not enforce NetworkPolicy — covered in the traps along with how to switch to Calico when you get to that topic.

Before you create the VMs

Get these right at creation time. Some cannot be changed afterwards.

IP forwarding on, on both VMs. GCE does not let you change it once the instance exists; if it is off, you delete the VM and start again.
x86-64 image, not Arm.
The same network tag on both machines, for example k8s.
A firewall rule for internal traffic with source ranges covering the node subnet (10.128.0.0/9 on the default network) and the pod CIDR 10.244.0.0/16. If you will switch to Calico later, add 192.168.0.0/16 now and save yourself the edit.
A firewall rule for NodePorts, TCP 30000-32767.

kubeadm names each node after its hostname. Check it:

hostname

GCE sets it from the instance name, so it should already say cp or worker.

Type `hostname` after every SSH

Not once. Every time. The first mistake in my error log is running kubeadm init on the worker instead of the control plane, and redoing it from scratch. In the exam the equivalent habit is switching context at the start of every task — doing the right thing on the wrong target scores zero.

Step 1 — Prepare both nodes

Run everything in this step on both cp and worker.

Swap, kernel modules, sysctl

kubelet will not run with swap on, and the pod network needs bridged traffic to pass through iptables.

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf
sudo modprobe overlay
sudo modprobe br_netfilter
printf 'net.bridge.bridge-nf-call-iptables=1\nnet.bridge.bridge-nf-call-ip6tables=1\nnet.ipv4.ip_forward=1\n' \
  | sudo tee /etc/sysctl.d/k8s.conf
sudo sysctl --system

Check it took:

swapon --show
lsmod | grep br_netfilter
sysctl net.ipv4.ip_forward

swapon --show must print nothing, and ip_forward must be 1.

Why `br_netfilter` matters more for Flannel

Flannel's VXLAN backend relies on bridged traffic going through iptables. Without this module, pods on the same node talk fine and pods on different nodes do not — which looks exactly like a firewall problem and sends you looking in the wrong place.

containerd

sudo apt-get update
sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

The SystemdCgroup line is the one people forget. Without it kubelet starts, runs erratically, and dies with logs that point everywhere except here. Confirm it:

sudo grep SystemdCgroup /etc/containerd/config.toml
sudo systemctl is-active containerd

kubeadm, kubelet and kubectl 1.33

sudo apt-get install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

kubeadm version
kubelet --version

Both should report v1.33.x. The apt-mark hold stops a routine apt upgrade from moving the version under you — and it is exactly what you will undo later, in the cluster upgrade post.

Step 2 — Bring up the control plane

Everything in this step runs on cp only.

Initialise

If you want to use kubectl from your own laptop later, the API server certificate needs the external IP in it. Look it up from the metadata server:

curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip; echo

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=<EXTERNAL_IP_CP>

Add the extra SAN now or regenerate certificates later

Leaving --apiserver-cert-extra-sans out is harmless until you try to reach the cluster from outside, and then the only fix is reissuing the API server certificate. Adding it costs nothing.

kubeadm ends by printing a kubeadm join command. Copying it is convenient but not required — you can regenerate it in step 3.

Give yourself a kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes

cp shows NotReady. That is correct: there is no pod network yet.

Check the pod CIDR before you install a CNI

This is the step I added after breaking the cluster, and it takes five seconds:

kubectl get node cp -o jsonpath='{.spec.podCIDR}'; echo

It must print 10.244.0.0/24. Anything else means kubeadm init ran with a different CIDR than the CNI you are about to install expects. Stop here and fix it — see the CIDR trap — because discovering it after the CNI is running takes twice as long.

Install Flannel

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl -n kube-flannel get pods -o wide -w

Stop watching with Ctrl+C once the pod is Running.

Pin the version when you practise rebuilds

latest is convenient for a first build, but it can change between two rebuilds, and then two identical runs give different results for no visible reason. Once you are rebuilding from memory, swap latest for a specific release tag so every run is the same.

Now confirm the network is actually up:

kubectl get nodes
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-proxy

cp should be Ready and CoreDNS Running. CoreDNS is the honest signal here: it stays Pending until the pod network genuinely works.

kube-proxy should be running too. Flannel leaves it in place — worth knowing, because kube-proxy problems are part of the cluster troubleshooting domain and you want one to practise on.

Step 3 — Join the worker

On cp, print a fresh join command. Tokens expire after 24 hours, so this is also how you rejoin a node on day two:

kubeadm token create --print-join-command

Run what it prints on worker, with sudo in front:

sudo kubeadm join <CP_INTERNAL_IP>:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

Back on cp:

kubectl get nodes -o wide
kubectl -n kube-flannel get pods -o wide

Both nodes Ready, both on v1.33.x, and now two Flannel pods — one per node.

One Flannel pod is not a Flannel bug

Flannel runs as a DaemonSet, so the pod count must equal the node count. Seeing only one means worker has not joined, not that the network is broken. On day two I read it as a broken Flannel.

Step 4 — Prove it

A green kubectl get nodes does not prove pods on different machines can reach each other. This is the test that does.

Start with a deployment:

kubectl create deployment web --image=nginx:1.27-alpine --replicas=4
kubectl expose deployment web --port=80
kubectl rollout status deployment/web
kubectl get pods -o wide

Every pod lands on worker. That is correct — the control plane carries the taint node-role.kubernetes.io/control-plane:NoSchedule and refuses ordinary workloads. It also means this test has not yet crossed between nodes at all.

Remove the taint temporarily so pods can run on both:

kubectl taint node cp node-role.kubernetes.io/control-plane:NoSchedule-
kubectl scale deployment web --replicas=6
kubectl get pods -o wide

The trailing - removes a taint rather than adding one. Pods should now sit on both nodes. Grab the IP of one on cp:

kubectl get pods --field-selector spec.nodeName=cp -o jsonpath='{.items[0].status.podIP}'; echo

Then call it from a throwaway pod pinned to worker:

kubectl run tmp --image=busybox --rm -it --restart=Never \
  --overrides='{"spec":{"nodeName":"worker"}}' -- wget -qO- --timeout=3 <POD_IP_ON_CP>

nginx HTML back means cross-node networking works. A hang and a timeout means packets are not crossing — check the firewall rule includes 10.244.0.0/16, that br_netfilter is loaded on both nodes, and that IP forwarding was on when the VMs were created.

You run kubectl on `cp` even when the pod runs on `worker`

--overrides pins the pod to worker, but the command still goes to the API server from wherever you are. I assumed it had to be typed on worker — the same wrong-machine instinct as before. SSH to a worker only for things that live on the machine: systemctl, journalctl, crictl.

Put the taint back and clean up:

kubectl taint node cp node-role.kubernetes.io/control-plane:NoSchedule
kubectl delete deployment web
kubectl delete svc web

Traps

Every one of these happened. Dates are from the error log.

Running a command on the wrong machine

Four separate mistakes in my log share one cause: not knowing which machine I was on. Running kubeadm init on worker. Reading The connection to the server localhost:8080 was refused as a broken cluster. Assuming a pod pinned to worker needed the command typed on worker.

The localhost:8080 one deserves its own line: it is never the cluster. It means kubectl found no kubeconfig — you are on a machine without one, or KUBECONFIG points somewhere wrong.

The CNI and the pod CIDR are one decision

On day one I built with Calico, which expects 192.168.0.0/16. On day two I rebuilt with Flannel — and copied the kubeadm init line from my own day-one notes, CIDR included. Flannel expects 10.244.0.0/16, and the result was:

failed to acquire lease: subnet "10.244.0.0/16" ... doesn't contain "192.168.0.0/24" PodCIDR

CNI	`--pod-network-cidr`
Flannel	`10.244.0.0/16`
Calico	`192.168.0.0/16`

Choose them together, never separately. The .spec.podCIDR check in step 2 exists because of this.

If you have already initialised with the wrong range, you do not need to reset. Patch Flannel to match the cluster instead:

curl -sLO https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
sed -i 's#10\.244\.0\.0/16#<YOUR_POD_CIDR>#g' kube-flannel.yml
grep -A6 'net-conf.json' kube-flannel.yml

Read the output before applying it:

kubectl apply -f kube-flannel.yml
kubectl -n kube-flannel rollout restart ds/kube-flannel-ds

Then update the GCP firewall rule to the new range, or pods on different nodes will not see each other — a symptom that looks identical to a forwarding problem.

Calico's eBPF manifest removes kube-proxy

On day one I installed Calico with custom-resources-bpf.yaml. It enables the eBPF dataplane, which replaces kube-proxy entirely — so there was no kube-proxy left to troubleshoot. For exam practice that is the wrong trade. Use custom-resources.yaml.

Both of these came from the same habit: reusing a procedure written for a different setup without re-checking the parameters that travel with it. My rule since: where the docs offer a more modern option than the default, take the default. I am training for an exam, not tuning a production cluster.

Flannel accepts NetworkPolicy and ignores it

Flannel handles networking only; it has no policy controller. You can kubectl apply a NetworkPolicy, kubectl get netpol will list it — and no traffic is blocked. No error, no warning.

Practising NetworkPolicy on Flannel is practising against an illusion. Before that topic, rebuild with Calico — see switching to Calico below.

`sudo` refusing to work on GCP

Most of my first day went here: sudo failing intermittently, which I read as a missing permission and chased through OS Login settings. It was not. When it came back on day two, disconnecting and reconnecting the SSH session fixed it. Only consider the permission angle if it survives a reconnect.

When it breaks

Symptom	Likely cause	Fix
`failed to acquire lease ... doesn't contain ... PodCIDR`	Init CIDR does not match Flannel's `net-conf.json`	Patch the manifest, or re-init with `10.244.0.0/16`
Only one Flannel pod	`worker` has not joined	Step 3
Pods on different nodes cannot reach each other	Firewall missing the pod CIDR, or IP forwarding off at creation	Fix the firewall rule; recreate the VM if forwarding was off
Same-node pods fine, cross-node broken	`br_netfilter` not loaded	Redo the kernel module step
Node stuck `NotReady`	No CNI, or Flannel crash-looping	`kubectl -n kube-flannel logs -l app=flannel --tail=30`
CoreDNS stuck `Pending`	Pod network not working	Same Flannel logs
NetworkPolicy applied, nothing blocked	Flannel has no policy controller	Rebuild with Calico
kubelet restarting constantly	`SystemdCgroup = true` missing	Fix `/etc/containerd/config.toml`, restart containerd
`kubeadm init` fails the CPU preflight	Fewer than 2 vCPUs	Use `e2-medium` or larger
`kubeadm init` fails on swap	Swap still on	Redo the swap step
Join says the token expired	Older than 24 hours	`kubeadm token create --print-join-command`
`localhost:8080 refused`	No kubeconfig on this machine	Check `hostname` and `KUBECONFIG`

Rebuild it

One build is not the point. Cluster installation and upgrades are in scope, and in the exam you will have the docs but not these notes — so plan to tear it down and rebuild at least twice without looking at them.

Two ways back to a clean slate:

A GCP snapshot — fastest. Take it after step 1, before kubeadm init. That is the most useful restore point: everything installed, nothing initialised.

kubeadm reset — the way the exam would expect. On both nodes:

sudo kubeadm reset -f
sudo rm -rf /etc/cni/net.d $HOME/.kube
sudo iptables -F
sudo iptables -t nat -F

Then start again from step 2.

Switching CNI? Remove the old interfaces too

A leftover config in /etc/cni/net.d makes kubelet pick up the wrong CNI. When moving between Flannel and Calico, also remove the old interfaces:

sudo ip link delete flannel.1 2>/dev/null
sudo ip link delete cni0 2>/dev/null

For Calico the interfaces are vxlan.calico and tunl0.

I did not time my first two builds — most of the first day went to the sudo problem, not to Kubernetes. Time the rebuilds instead; they are the number that matters.

Switching to Calico for NetworkPolicy

Only two things change: the CIDR at init, and the CNI install. Everything else is the same.

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-cert-extra-sans=<EXTERNAL_IP_CP>

Install Calico — v3.32.1 is the version I ran against 1.33, following the Tigera self-managed install guide:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/v1_crd_projectcalico_org.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/custom-resources.yaml

That last file is custom-resources.yaml, not custom-resources-bpf.yaml. Confirm the dataplane is not eBPF:

kubectl get installation default -o jsonpath='{.spec.calicoNetwork.linuxDataplane}'; echo

Empty or Iptables is right. Then wait for Calico to settle, and check both Calico and kube-proxy are running:

watch kubectl get tigerastatus
kubectl get pods -n calico-system
kubectl -n kube-system get pods -l k8s-app=kube-proxy

apiserver, calico and ippools should all show AVAILABLE=True. Remember to open 192.168.0.0/16 in the firewall rule.

Optional: kubectl from your laptop

Handy while learning workloads. Open TCP 6443 to your own IP only — never 0.0.0.0/0:

curl -s ifconfig.me; echo

Create a firewall rule with target tag k8s, source range <YOUR_IP>/32, allowing TCP 6443. Then copy the kubeconfig over and point it at the external IP (the sed -i '' form is macOS):

scp cp:~/.kube/config ~/.kube/config-cka
sed -i '' "s#server: https://.*#server: https://<EXTERNAL_IP_CP>:6443#" ~/.kube/config-cka
export KUBECONFIG=~/.kube/config-cka
kubectl get nodes

Do not get too comfortable with it. In the exam you work on the node's own terminal, and cluster troubleshooting means SSH-ing into real machines.

Stop paying for it

gcloud compute instances stop cp worker

Stopped instances stop billing for CPU and memory, but disks keep billing. If you are done for a while, delete them — and deleting and rebuilding is exactly the practice described above.

The docs pages

These are on kubernetes.io, which you can open during the exam. Knowing where they are is part of the skill.

Installing kubeadm
Creating a cluster with kubeadm
Container runtimes — the containerd and cgroup driver details
kubeadm reset
Taints and tolerations

Next: Set up your terminal for the exam · Back to the CKA roadmap