20. November 2020 13 min read
Traefik v2 and LetsEncrypt cert-manager on RaspberryPi4 kubernetes cluster
Most of the times you just want to simply transfer your simple webpage to your raspberry pi cluster at home. The webpage is of course running on https and you are obtaining free certificates from LetsEncrypt using certbot in reality. This blog post guides you through some pitfalls I encountered while doing my own setup. My setup is Kubernetes 1.19.2, on mixed Raspberry Pi 4 and Raspberry Pi 3 cluster, both with 32 bit arm operating systems (Raspbian).
I will skip basic installation as this point assumes you already have a cluster up and running, but you live without LoadBalancer since Traefik replaces load balancers like MetalLB, so you do not need one. We will use Helm 3 to install stuff on our cluster as that is easy and fast way, but we will have to modify some of the values to get it working nicer. Both Traefik and cert-manager have great installation guides, so I will skip the commands for setting up the repo like helm repo add ... and will drop directly to the changes in values and update/install commands.
Traefik v2 and http
Traefik moved to v2 and while migration is not that straight forward as some features are still missing from Traefik Helm chart it is useful to mention that it does support ACME Certificates (although with some minor tweaks). The simple exercise here will be to expose the Traefik Dashboard to internet (just for test, do not keep it exposed, but use basicAuth module to protect it with username and password) via https. Let`s first check the values we need to change, and below I will write some notes that I have observed and the problems that I needed to overcome.
# traefik2-helm-values.yaml # ... additionalArguments: - "--certificatesresolvers.letsencrypt.acme.email=youremail" - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json" - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory" - "--certificatesResolvers.letsencrypt.acme.tlsChallenge=true" - "--providers.kubernetesingress.ingressclass=traefik-cert-manager" # ... # Configure ports ports: # The name of this one can't be changed as it is used for the readiness and # liveness probes, but you can adjust its config to your liking traefik: port: 9000 # ... Keep in mind we want to expose this ourselves not automatically and on master node 443 port expose: false # The exposed port for this service exposedPort: 9000 # The port protocol (TCP/UDP) protocol: TCP # ... service: enabled: true type: LoadBalancer # ... externalIPs: - your.master.node.host.ip.which.is.exposed.to.internet
Above commands set ingressclass to traefik-cert-manager (which will be cert-manager`s default label below) and enables tls challenge. I doubt Letsencrypt stuff is needed since from here on cert-manager takes over, but I left it in for completeness sake. After that we do not want to expose dashboard port 9000 to outside world, but we will rather write our own ingressroute to expose it on standard https port of 443. Next we enable LoadBalancer feature of Traefik and set the externalIP to your master`s node external IP, which is exposed to internet as either DMZ or normal port forwarding on your router. Keep in mind that this should not be mistaken as your public IP in most cases (unless your RPI directly connects to internet).
With this all set, you can run the helm command
helm upgrade traefik traefik/traefik --values traefik2-helm-values.yaml --namespace kube-system
Now lets expose the dashboard to internet ourselves via simple service and IngressRoute. Just note, that this is the same way you can expose any other pod out there
# traefik-dashboard.yaml kind: Service apiVersion: v1 metadata: labels: k8s-app: treafik name: treafik-dashboard namespace: kube-system spec: ports: - name: dashboard port: 9000 targetPort: 9000 selector: app.kubernetes.io/instance: traefik app.kubernetes.io/name: traefik --- apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: treafik-dashboard-ingress namespace: kube-system spec: entryPoints: - web routes: - match: Host(`example.com`) kind: Rule services: - name: treafik-dashboard kind: Service port: 9000
Above yaml defines a Service called traefik-dashboard, which links to pods port 9000 and then creates IngressRoute, which tells Traefik to direct all traffic coming to entryPoint web with host example.com to service traefik-dashboard and port 9000.
Now lets apply this to our cluster with kubectl
kubectl apply -f traefik-dashboard.yaml
Now your browser should be serving the Traefik Dasbhoard on HTTP of your host. Some common things you could miss is like setup of your DNS server to point to your Kubernetes cluster public IP or forwarding some ports (we only need 80 and 443) on routers to your kubernetes master. Keep in mind that 9000 is clusters internal port in this case and that only http and https ports are exposed. If you get User forbidden error then you should create a clusterrolebinding
kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin
Now we want to put more secure https protocol to our dashboard and for that our above Helm values already have prepared Traefik for all we need, but we still need to install Certificate manager (cert-manager) to ensure certificates get requested, renewed and basically managed. Cert-manager is the right tool for this job and there is almost no change needed to original helm chart of it to get it working except for one flag:
kubectl create namespace cert-manager helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true
Now cert manager should be running as pod on your cluster, but it would idle. First we need to set the clusterIssuer to LetsEncrypt (described also in documentation, but lets repeat here).
# cluster-issuer.yaml apiVersion: cert-manager.io/v1alpha2 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email: email@example.com server: https://acme-staging-v02.api.letsencrypt.org/directory privateKeySecretRef: # Secret resource used to store the account's private key. name: issuer-account-key-for-something-very-elaborate-you-dont-know # Add a single challenge solver, HTTP01 solvers: - http01: ingress: class: traefik-cert-manager --- apiVersion: cert-manager.io/v1alpha2 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email: firstname.lastname@example.org server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: # Secret resource used to store the account's private key. name: issuer-account-key-for-something-very-elaborate-you-dont-know # Add a single challenge solver, HTTP01 solvers: - http01: ingress: class: traefik-cert-manager
This will install both production and staging ClusterIssuers to your site and I suggest you start playing with Staging, because that has higher limits and more slack for mistakes. It will require however to manually check the red lock's reason to see if certificate was actually issued. Now lets apply this file with kubectl
kubectl apply -f cluster-issuer.yaml
This means Cert-manager is ready on our cluster and prepared to request Certificates for our domain.
Expose our dashboard through https with Traefik v2
Now lets create the Certificate for our dashboard
# traefik-dashboard-certificate.yaml apiVersion: cert-manager.io/v1alpha2 kind: Certificate metadata: name: traefik-dashboard-cert namespace: kube-system spec: commonName: example.com secretName: example-cert dnsNames: - example.com issuerRef: name: letsencrypt-staging kind: ClusterIssuer
Above snippet will create a traefik-dashboard-cert with secret name example-cet on domain name example.com (be careful commonName and dnsNames need to match using LetsEncrypt Staging (issuerRef). Now lets apply this file, so that cert-manager actually starts requesting our certificate.
kubectl apply -f traefik-dashboard-certificate.yaml
At this point you should see certificate on your cluster
kubectl get certificates -A
Traefik actively monitors the IngressRoute, so we need to change just two items in our above snippet. That is entryPoint needs to point to websecure (that is https exposed port from Traefik by default) and tls needs to point to certificate.
# traefik-dashboard.yaml apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: treafik-dashboard-ingress namespace: kube-system spec: entryPoints: - websecure # ... tls: secretName: example-cert
After you changed above values, lets apply this to our cluster with kubectl again
kubectl apply -f traefik-dashboard.yaml
Tadaaa. Now your site should be accessible through https with a red lock because of LetsEncrypt staging certificate. But it works. If it does not, I suggest you look at logs from cert-manager pod to find suggestions and describe challenges to get more details
Change your Certificate issuerRef to letsencrypt production
Now to get a nice green lock of https security in your browser you need to change your certificate issuerRef in traefik-dashboard-certificate.yaml to letsencrypt-prod.
# traefik-dashboard-certificate.yaml apiVersion: cert-manager.io/v1alpha2 kind: Certificate metadata: name: traefik-dashboard-cert namespace: kube-system spec: commonName: example.com secretName: example-cert dnsNames: - example.com issuerRef: name: letsencrypt-prod kind: ClusterIssuer
Now I like to cleanup before I apply new certificate as I had some problems while setting this up, so before you apply make sure you delete existing certificate and its secret, before re-applying the changed certificate
kubectl delete certificate -n kube-system traefik-dashboard-cert kubectl delete secret -n kube-system example-cert kubectl apply -f traefik-dashboard-certificate.yaml
After few minutes you should have a green lock (you might have to reopen the browser) to refresh the certificate and remove the allowed exception.