17. March 2020

Errors while deploying weave on kubernetes

While Kubernetes cluster deployment to the Raspberry Pi4 or even Raspberry Pi3 clusters (or mixed) sounds very easy, I am always amazed how much knowledge I actually lack or even forget in between successful deployments. That also comes at cost when trying to apply update. Below is just a log of information which helped me debug the following error messages:

kubelet[25358]: W0317 25358 cni.go:202] Error validating CNI config &{weave 0.3.0 false [0x78b3a00 0x78b3a40] [123 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 48 34 44 10 32 32 32 32 34 110 97 109 101 34 58 32 34 119 101 97 118 101 34 44 10 32 32 32 32 34 112 108 117 103 105 110 115 34 58 32 91 10 32 32 32 32 32 32 32 32 123 10 32 32 32 32 32 32 32 32 32 32 32 32 34 110 97 109 101 34 58 32 34 119 101 97 118 101 34 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 119 101 97 118 101 45 110 101 116 34 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 104 97 105 114 112 105 110 77 111 100 101 34 58 32 116 114 117 101 10 32 32 32 32 32 32 32 32 125 44 10 32 32 32 32 32 32 32 32 123 10 32 32 32 32 32 32 32 32 32 32 32 32 34 116 121 112 101 34 58 32 34 112 111 114 116 109 97 112 34 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 32 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 32 116 114 117 101 125 44 10 32 32 32 32 32 32 32 32 32 32 32 32 34 115 110 97 116 34 58 32 116 114 117 101 10 32 32 32 32 32 32 32 32 125 10 32 32 32 32 93 10 125 10]}: [failed to find plugin "weave-net" in path [/opt/cni/bin]]
kubelet[25358]: W0317 25358 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d

Looking at kubectl output you usually get:

$ kubectl get pods -n kube-system -l name=weave-net -o wide
NAME              READY   STATUS             RESTARTS   AGE    IP              NODE                  NOMINATED NODE   READINESS GATES
weave-net-4rmwp   1/2     CrashLoopBackOff   34         104m   192.168.0.192   raspberrypi4-master              
weave-net-fw2vx   2/2     Running            1          3h9m   192.168.0.157   raspberrypi3-node                
weave-net-fx9vr   2/2     Running            0          10h    192.168.0.90    raspberrypi4-node                

What is more surprising in my example is that the crashloopBack was actually on master, so nothing was working. Of course I went over the issues like Github Kubernetes/kubeadm 1704 and most useful GitHub Kubernetes issue 54918 where some advices are there, but they are either strange (777 of some directory) or did not work for me (creating custom conflist). But since that is the first hit on Google for the:

 Unable to update cni config: no valid networks found in /etc/cni/net.d

I seem to always read it again and understand the problem. In reality RTFM (Read the f***ing manual is the only way to go and there clearly tells you to do a proper cleanup on your master node which is:

# remove current deployment (be careful if you just updated - then version here is new, but old one needs to be removed
$ kubectl delete -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
# wait that all nodes terminate
# $ kubectl get pods -n kube-system -l name=weave-net -o wide
# now follow RTFM
$ weave reset
$ sudo rm /opt/cni/bin/weave-*
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Now weave starts as intended.

To run the weave command you need to first install debug version of weave

Newest from this category: