Self-signed certificates on MutatingWebhooks require double-encryption

I’m using Cybozu’s TopoLVM to provide local LVM-based storage to a bare-metal Kubernetes cluster, in an intelligent fashion.

I’m also using Jetstack’s cert-manager with the --namespace argument, to watch for certificate resources in a particular namespace only, so I wasn’t able to use cert-manager with TopoLVM, which is normally a pre-requisite.

The deployment docs tell me that I can avoid cert-manager if I use a self-signed certificate for the TopoLVM mutatingwebhook, which I thought wouldn’t be too difficult. I ran the following to generate the necessary cert, key, and cacert (valid for 100 years):

openssl genrsa -out rootCA.key 4096

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 35600 -out rootCA.crt

openssl genrsa -out controller.topolvm-system.svc.key 2048 

openssl req -new -sha256 -days 36500  -key controller.topolvm-system.svc.key -subj '/CN=controller.topolvm-system.svc' -out controller.topolvm-system.svc.csr

openssl x509 -req -in controller.topolvm-system.svc.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out controller.topolvm-system.svc.crt -days 36500 -sha256

The documentation said to add the caBundle field to the mutatingwebhook YAML in PEM format, so I initially added the entire rootCA.crt, including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----. This failed due to bad base64 encoding, so I removed the BEGIN and END lines, and the PEM data from the certificate was accepted as valid base64.

However, the webhook wasn’t trusted, and any pods I deployed failed with messages about certificate signed by unknown authority.

Turns out what was required was to base64-encode the PEM file, and paste the resulting base64-encoded string into caBundle. I.e, I set caBundle to the output of cat rootCA.crt | base64.

I found some kindred fellow-sufferers, whose confusion and eventual frustration echo my own!