Customizing Istio's trace sampling rate per-pod

We're using Istio across an increasing range of services, and occasionally we need to "dial up" the default sampling rate from 1% to 100% for debug / analysis reasons (we send our traces to Jaeger).

I wanted to know whether it's possible to change sampling rate for some pods/namespaces, without changing the sampling rate globally (we'd receive far too many traces in this case!)

There's a thread over at https://github.com/istio/istio/issues/12818 which touches on this requirement, but as of Aug 2021, it hasn't come to an easy conclusion.

However, I found an elegant-but-slightly-convoluted method to achieve the custom sampling rate, using pod annotations...

Per https://istio.io/docs/reference/config/istio.mesh.v1alpha1/#ProxyConfig, it's possible to alter the Istio proxy config by applying an annotation... so I applied this:

  Annotations:      proxy.istio.io/config: |
                      tracing:
                        sampling: 100
                        zipkin:
                          address: jaeger-collector.jaeger:9411 

Note that it's not enough to simply pass the sampling value - you have to pass the entire tracing config, including the zipkin destination address. I add the following annotations to my application helm charts, so that all I have to do to ramp up samping is to pass --set samplingRate=100:

      annotations:
        proxy.istio.io/config: |
          tracing:
            sampling: {{ .Values.samplingRate }}
            zipkin:
              address: jaeger-collector.jaeger:9411

In conclusion, this approach does require redeploying pods, but provided you're using multiple replicas, this can be achieved with no noticable impact, and you can dial sampling up/down as required!