Applying PSPs to istio-cni-node pods
I work on Kubernetes cluster which are a whole lot more locked-down than typical installation instructions / application defaults would suggest. In one such cluster, we use PodSecurityPolicies to apply a minimal set of privileges to each pod, and make exceptions on a case-by-case basis.
On the same cluster, we use the Istio service mesh to secure traffic between our pods using mutualTLS. We take advantage of Istio's CNI plugin to allow the Istio sidecar to inject the "traffic interception" rules when pods start up, without requiring privileged access for every pod with a sidecar.
The CNI plugin creates a daemonset (a pod per node), which requires privileged access to inject the interception rules. Our default, restrictive PSP policy prevents these istio-cni-node pods from ever starting though, as illustrated below:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 114s (x17 over 7m22s) daemonset-controller Error creating: pods "istio-cni-node-" is forbidden: unable to validate against any pod security policy: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0].hostPath.pathPrefix: Invalid value: "/opt/cni/bin": is not allowed to be used spec.volumes[1].hostPath.pathPrefix: Invalid value: "/etc/cni/net.d": is not allowed to be used]The error above is pointing out that PSPs (quite rightly) prevented an arbitrary pod from mounting critical host directories, and having its way with them.
In this case, access to /opt/cni/bin and /etc/cni/net.d is a requirement for using Istio CNI (and the alternative of allowing every pod privileged access is much worse!), so we deploy a PSP, ClusterRole, and ClusterRoleBinding as illustrated below (you can grab a copy here):
| apiVersion: policy/v1beta1 | |
| kind: PodSecurityPolicy | |
| metadata: | |
| name: istio-cni-node | |
| annotations: | |
| seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default,runtime/default' | |
| apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' | |
| seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' | |
| apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' | |
| spec: | |
| allowedHostPaths: | |
| - pathPrefix: "/opt/cni/bin" | |
| - pathPrefix: "/etc/cni/net.d" | |
| privileged: false | |
| allowPrivilegeEscalation: true | |
| defaultAllowPrivilegeEscalation: false | |
| requiredDropCapabilities: | |
| - ALL | |
| # Allow non-persistent volume types. | |
| volumes: | |
| - 'configMap' | |
| - 'emptyDir' | |
| - 'projected' | |
| - 'secret' | |
| - 'downwardAPI' | |
| - 'hostPath' | |
| hostNetwork: true | |
| hostIPC: false | |
| hostPID: false | |
| runAsUser: | |
| # Istio containers don't run as non-root, so until this is fixed, we have to permit this | |
| rule: 'RunAsAny' | |
| seLinux: | |
| # This policy assumes the nodes are using AppArmor rather than SELinux. | |
| rule: 'RunAsAny' | |
| supplementalGroups: | |
| rule: 'RunAsAny' | |
| fsGroup: | |
| rule: 'RunAsAny' | |
| readOnlyRootFilesystem: false | |
| --- | |
| # Cluster role which grants access to the default pod security policy | |
| apiVersion: rbac.authorization.k8s.io/v1 | |
| kind: ClusterRole | |
| metadata: | |
| name: psp-istio-cni-node | |
| rules: | |
| - apiGroups: | |
| - policy | |
| resourceNames: | |
| - istio-cni-node | |
| resources: | |
| - podsecuritypolicies | |
| verbs: | |
| - use | |
| --- | |
| # Cluster role binding for default pod security policy granting all authenticated users access | |
| apiVersion: rbac.authorization.k8s.io/v1 | |
| kind: RoleBinding | |
| metadata: | |
| name: psp-istio-cni-node | |
| namespace: istio-system | |
| roleRef: | |
| apiGroup: rbac.authorization.k8s.io | |
| kind: ClusterRole | |
| name: psp-istio-cni-node | |
| subjects: | |
| - apiGroup: rbac.authorization.k8s.io | |
| kind: Group | |
| name: system:nodes | |
| # For all service accounts in the kube-system namespace | |
| - kind: ServiceAccount | |
| name: istio-cni |