20. November 2020 13 min read

Traefik v2 and LetsEncrypt cert-manager on RaspberryPi4 kubernetes cluster

Most of the times you just want to simply transfer your simple webpage to your raspberry pi cluster at home. The webpage is of course running on https and you are obtaining free certificates from LetsEncrypt using certbot in reality. This blog post guides you through some pitfalls I encountered while doing my own setup. My setup is Kubernetes 1.19.2, on mixed Raspberry Pi 4 and Raspberry Pi 3 cluster, both with 32 bit arm operating systems (Raspbian).
I will skip basic installation as this point assumes you already have a cluster up and running, but you live without LoadBalancer since Traefik replaces load balancers like MetalLB, so you do not need one. We will use Helm 3 to install stuff on our cluster as that is easy and fast way, but we will have to modify some of the values to get it working nicer. Both Traefik and cert-manager have great installation guides, so I will skip the commands for setting up the repo like helm repo add ... and will drop directly to the changes in values and update/install commands.

Traefik v2 and http

Traefik moved to v2 and while migration is not that straight forward as some features are still missing from Traefik Helm chart it is useful to mention that it does support ACME Certificates (although with some minor tweaks). The simple exercise here will be to expose the Traefik Dashboard to internet (just for test, do not keep it exposed, but use basicAuth module to protect it with username and password) via https. Let`s first check the values we need to change, and below I will write some notes that I have observed and the problems that I needed to overcome.

# traefik2-helm-values.yaml
# ...
additionalArguments:
  - "--certificatesresolvers.letsencrypt.acme.email=youremail"
  - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
  - "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
  - "--certificatesResolvers.letsencrypt.acme.tlsChallenge=true"
  - "--providers.kubernetesingress.ingressclass=traefik-cert-manager"
# ...
# Configure ports
ports:
  # The name of this one can't be changed as it is used for the readiness and
  # liveness probes, but you can adjust its config to your liking
  traefik:
    port: 9000
    # ... Keep in mind we want to expose this ourselves not automatically and on master node 443 port
    expose: false
    # The exposed port for this service
    exposedPort: 9000
    # The port protocol (TCP/UDP)
    protocol: TCP
# ...
service:
  enabled: true
  type: LoadBalancer
  # ...
  externalIPs:
    - your.master.node.host.ip.which.is.exposed.to.internet

Above commands set ingressclass to traefik-cert-manager (which will be cert-manager`s default label below) and enables tls challenge. I doubt Letsencrypt stuff is needed since from here on cert-manager takes over, but I left it in for completeness sake. After that we do not want to expose dashboard port 9000 to outside world, but we will rather write our own ingressroute to expose it on standard https port of 443. Next we enable LoadBalancer feature of Traefik and set the externalIP to your master`s node external IP, which is exposed to internet as either DMZ or normal port forwarding on your router. Keep in mind that this should not be mistaken as your public IP in most cases (unless your RPI directly connects to internet).

With this all set, you can run the helm command

helm upgrade traefik traefik/traefik --values traefik2-helm-values.yaml --namespace kube-system

Now lets expose the dashboard to internet ourselves via simple service and IngressRoute. Just note, that this is the same way you can expose any other pod out there

# traefik-dashboard.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: traefik
  name: traefik-dashboard
  namespace: kube-system
spec:
  ports:
    - name: dashboard
      port: 9000
      targetPort: 9000
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-ingress
  namespace: kube-system
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`example.com`)
      kind: Rule
      services:
        - name: traefik-dashboard
          kind: Service
          port: 9000

Above yaml defines a Service called traefik-dashboard, which links to pods port 9000 and then creates IngressRoute, which tells Traefik to direct all traffic coming to entryPoint web with host example.com to service traefik-dashboard and port 9000.

Now lets apply this to our cluster with kubectl

kubectl apply -f traefik-dashboard.yaml

Now your browser should be serving the Traefik Dasbhoard on HTTP of your host. Some common things you could miss is like setup of your DNS server to point to your Kubernetes cluster public IP or forwarding some ports (we only need 80 and 443) on routers to your kubernetes master. Keep in mind that 9000 is clusters internal port in this case and that only http and https ports are exposed. If you get User forbidden error then you should create a clusterrolebinding

kubectl create clusterrolebinding --user system:serviceaccount:kube-system:default kube-system-cluster-admin --clusterrole cluster-admin

Cert-manager

Now we want to put more secure https protocol to our dashboard and for that our above Helm values already have prepared Traefik for all we need, but we still need to install Certificate manager (cert-manager) to ensure certificates get requested, renewed and basically managed. Cert-manager is the right tool for this job and there is almost no change needed to original helm chart of it to get it working except for one flag:

kubectl create namespace cert-manager
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.3 --set installCRDs=true

Now cert manager should be running as pod on your cluster, but it would idle. First we need to set the clusterIssuer to LetsEncrypt (described also in documentation, but lets repeat here).

# cluster-issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: your@email.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: issuer-account-key-for-something-very-elaborate-you-dont-know
    # Add a single challenge solver, HTTP01
    solvers:
      - http01:
          ingress:
            class: traefik-cert-manager
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: your@email.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: issuer-account-key-for-something-very-elaborate-you-dont-know
    # Add a single challenge solver, HTTP01
    solvers:
      - http01:
          ingress:
            class: traefik-cert-manager

This will install both production and staging ClusterIssuers to your site and I suggest you start playing with Staging, because that has higher limits and more slack for mistakes. It will require however to manually check the red lock's reason to see if certificate was actually issued. Now lets apply this file with kubectl

kubectl apply -f cluster-issuer.yaml

This means Cert-manager is ready on our cluster and prepared to request Certificates for our domain.

Expose our dashboard through https with Traefik v2

Now lets create the Certificate for our dashboard

# traefik-dashboard-certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: traefik-dashboard-cert
  namespace: kube-system
spec:
  commonName: example.com
  secretName: example-cert
  dnsNames:
    - example.com
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer

Above snippet will create a traefik-dashboard-cert with secret name example-cet on domain name example.com (be careful commonName and dnsNames need to match using LetsEncrypt Staging (issuerRef). Now lets apply this file, so that cert-manager actually starts requesting our certificate.

kubectl apply -f traefik-dashboard-certificate.yaml

At this point you should see certificate on your cluster

kubectl get certificates -A

Traefik actively monitors the IngressRoute, so we need to change just two items in our above snippet. That is entryPoint needs to point to websecure (that is https exposed port from Traefik by default) and tls needs to point to certificate.

# traefik-dashboard.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-ingress
  namespace: kube-system
spec:
  entryPoints:
    - websecure
# ...
  tls:
    secretName: example-cert

After you changed above values, lets apply this to our cluster with kubectl again

kubectl apply -f traefik-dashboard.yaml

Tadaaa. Now your site should be accessible through https with a red lock because of LetsEncrypt staging certificate. But it works. If it does not, I suggest you look at logs from cert-manager pod to find suggestions and describe challenges to get more details

Change your Certificate issuerRef to letsencrypt production

Now to get a nice green lock of https security in your browser you need to change your certificate issuerRef in traefik-dashboard-certificate.yaml to letsencrypt-prod.

# traefik-dashboard-certificate.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: traefik-dashboard-cert
  namespace: kube-system
spec:
  commonName: example.com
  secretName: example-cert
  dnsNames:
    - example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Now I like to cleanup before I apply new certificate as I had some problems while setting this up, so before you apply make sure you delete existing certificate and its secret, before re-applying the changed certificate

kubectl delete certificate -n kube-system traefik-dashboard-cert
kubectl delete secret -n kube-system example-cert
kubectl apply -f traefik-dashboard-certificate.yaml

After few minutes you should have a green lock (you might have to reopen the browser) to refresh the certificate and remove the allowed exception.