Securing Kubernetes and Google Cloud with Workload Identity

5 min readMay 26, 2021

Have you ever thought about how your Kubernetes cluster consumes your Google services? At Devgurus (a DMI company) we started this discussion some days ago (thanks to the Day-2 Operations) and we discovered an elegant solution using Workload Identity.

Introduction

If you work in a containerized enterprise application orchestrated with Kubernetes (K8s), I’m pretty sure you faced issues and concerns about authentication and authorization, especially if you interact with other services. You can think this problem is easier to solve if the other services are in the same provider, e.g. resources like buckets or pub/subtopics in the same Google Cloud account. And it is simplified: the Google Application Default Credentials (ADC) will find your service account credentials following this algorithm:

1. If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set, ADC uses the service account key or configuration file that the variable points to.2. If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set, ADC uses the service account that is attached to the resource that is running your code.This service account might be a default service account provided by Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run, or Cloud Functions. It might also be a user-managed service account that you created.3. If ADC can't use any of the above credentials, an error occurs.

So if you didn’t configure any particular service account, your Kubernetes workload (alias your app deployed in a pod) will consume your cloud resources with a default K8s or Compute service account (PROJECT_ID@ compute.google.com). Then, in order to move forward, you just have to grant the proper permissions to this default account and, vualá! your application will be able to write magically with your topic without any credentials set.

However, this is an anti-pattern in enterprise applications because were not following the Least Privilege principle: Since then, all the applications deployed in your cluster will be able to interact with the granted resources. This can get even worse if you, trying to unblock the development, grant Owner permission to the default SA, the chaos is coming! (and this happens more than you can think).

To avoid this anti-pattern we’ve been using the GOOGLE_APPLICATION_CREDENTIALS environment variable describe in ADC:

1. Create a minimal service account and assign the role needed.
2. Deploy the pod using secrets to set the Google Application Credentials at the container level.

This solution is good and production ready but it has some complaints:

It’s confusing to have K8s service accounts, Google Cloud service accounts, Deployment service accounts and Execution service accounts.
The need to maintain a mapping between the SA account created in the infrastructure and the code itself.
Granularity: The role is set by default at the minimum level.
Multi-cluster and multi-namespace projects: If you like to differentiate permissions depending on “class” of the workload, some extra management tasks are needed.

GCP Docs: Identity sameness accessing Google Cloud APIs with Workload Identity

Using Workload Identity

The most elegant solution we found was using Workload Identity.
Google has a really good guide here, so we’re not going to just copy/paste what it’s explained there, this article aims to explain the key concepts and how we implemented it.

But first, we need to understand that when we work with Google Cloud and K8s we have Google service accounts (not human accounts used to access Google APIs) on one hand and the other hand Kubernetes service accounts (identities used to consume K8s own API).
This is important to clarify because a lot of developers when they review the pod configuration think the SA that must be granted is the one that appears in the yaml:

Google’s proposal is to add this K8s SA to an Identity pool to assign GCP roles and treat it like a GCP SA. For example, the previous K8s service account will be identified by Google as:

serviceAccount:commercetools-accelera7e39126c.svc.id.goog[default/auth-v1-helm-microservice-chart]"

where your $PROJECT_ID, $NAMESPACE, $KUBERNETES_SA will conform to this id.

This sounds promising, so let’s see how can we enable it:

1- Enable Workload Identity in the cluster

Using Terraform and the Google Cloud Platform Provider, you can enable workload identity easily just adding:

resource "google_container_cluster" "primary" {
...
  workload_identity_config {
      identity_namespace = "${var.project_id}.svc.id.goog"
  }
}

Btw, we have this publicly available as part of our Terraform module, created to bootstrap e-commerce applications, check it out.

2- Deploy the workload in K8s with a dedicated service account

We strongly recommend managing the packages in Kubernetes via Helm. That way we can assure that every new deployment has a dedicated SA, among other K8s resources like HPA, secrets…
The default Helm chart created with helm create CHART_NAME should work:

You can verify now that the workload associated with this pod has the proper service account:

$kubectl get pods --field-selector=spec.serviceAccountName=auth-v1-helm-microservice-chart
NAME                                               READY   STATUS   
auth-v1-helm-microservice-chart-754777d67f-c9bl6   2/2     Running   
auth-v1-helm-microservice-chart-754777d67f-qk27r   2/2     Running

3- Configure the K8s SA to act as a Google SA

At this moment, our application is already trying to connect to the Google APIs using the Workload Identity. If we output the email used by the Google SDK, we will see an id that includes the project key and some K8s infixes:

{ client_email: 'commercetools-accelera7e39126c.svc.id.goog' }

What’s remaining is to map this Workload Identity to an actual Google SA so we can assign roles or permissions. It’s important to note that the K8s SA has a name based on the K8s namespace, project id, and workload name:

gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:commercetools-accelera7e39126c.svc.id.goog[default/auth-v1-helm-microservice-chart]"   topic-publisher@commercetools-accelera7e39126c.iam.gserviceaccount.com

Finally, we annotate the K8s with this mapping:

kubectl annotate serviceaccount --namespace default auth-v1-helm-microservice-chart iam.gke.io/gcp-service-account=topic-publisher@commercetools-accelera7e39126c.iam.gserviceaccount.com

Conclusion

After all this configuration, that I know can be a bit overkill for just a few microservices, we end with a solution to authenticate and authorize workloads from your Kubernetes cluster into your Google Cloud services. This solution has some advantages over the explicit pod credentials setup:

Fine grained permissions set up
Avoid SA key generation and rotations
No extra environment variables added
Better understanding of the Least Privilege principle
DevSecOps compliant

Acknowledgments

I would like to thanks the whole Devgurus/DMI DevOps team, including the former members, as well as the development team which help to debug the initial permission errors: Santi, Lucas, Xema, Alberto…

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author 👇