The journey of secrets in Kubernetes at Zapier

Secrets management is a very important part of continuous deployment and infrastructure as code. At Zapier, we started adopting Kubernetes in late 2017. Back then, we used to use Helm and some custom scripts to deploy our applications to Kubernetes from Helm configs in a git repository. We used to use the Helm secrets plugin to manage our secrets in the Helm configs git repository. Learn why we changed from using the Helm secrets plugin to sops-secrets-operator and finally to Vault.

Helm secrets

In 2017, we started adopting Kubernetes at Zapier. At that time, there was not much stable tooling around to continuously deploy applications using Helm to Kubernetes. So, we wrote a few Python scripts to deploy applications from helm configs to Kubernetes. These scripts were wired to our CI/CD solution—which was then Jenkins—to run whenever changes got merged to the main branch of the Helm configs repo. To store secrets in our Helm configs in a secure way, we used a Helm plugin called helm-secrets to decrypt, edit and encrypt our secrets in the Helm configs git repository. The helm-secrets plugin uses SOPS under the hood to encrypt or decrypt secrets using various key providers like PGP, AWS or GCP KMS, etc.

Back in those days, many community Helm charts used to accept secrets in Helm config values. So, we followed the same pattern in our in-house Helm charts. With helm-secrets plugin, we could dump a part of the helm config values inside a secrets.yaml file and then encrypt it using helm-secrets before pushing it to git. We also used a tool called helm-wrapper as an alias for Helm CLI and it’d take care of automatically decrypting secrets.yaml file during application of helm charts from the Helm config files. The developer workflow with helm-secrets was very simple

# Edit secrets file and encrypt afterward
$ helm secrets edit path/to/app/cluster-01/secrets.yaml

The CI/CD tool had permission to decrypt the secrets in the git repo and apply the Helm chart for the applications to deploy changes to Kubernetes. This worked pretty well until we started exploring the use of external or managed CI/CD tools. In terms of good security practices, it’s not ideal for tools in the deployment pipeline to be aware of application secrets. With helm-secrets, the CI/CD tool needed to have permission to decrypt the secrets to deploy the application to Kubernetes.

Summary

Pros

• Simple and easy to use. Works out of the box with most upstream Helm charts.

Cons

• CI/CD tooling needs to have permissions to decrypt secrets

• Unencrypted secrets might get committed to repo by mistake

SOPS secrets operators

In 2018, we started migrating off from our custom Helm tooling to ArgoCD to continuously deploy our applications to Kubernetes from our git repositories. Initially, we had tried to customize ArgoCD images to replace the shipped helm binary with helm-wrapper along with helm-secrets plugin support. However, our initial attempts were not successful and we eventually found a better solution.

That’s when we came across sops-secrets-operator, which manages Kubernetes Secret Resources created from user-defined SopsSecret custom resource objects using sops. Here, we’d first create a SopsSecret custom resource object inside the templates directory of our application's Helm chart, something like

apiVersion: isindir.github.com/v1alpha3
kind: SopsSecret
metadata:
  name: example-sopssecret
spec:
  secretTemplates:
    - name: my-secret-name-1
      labels:
        label1: value1
      annotations:
        key1: value1
      stringData:
        data-name0: data-value0
      data:
        data-name1: ZGF0YS12YWx1ZTE=

Then, we’d encrypt the above SopsSecret YAML file with the help of sops using

$ sops -e --encrypted-suffix=secret_templates path/to/app/templates/sopssecrets.yaml

We’d then reference the secret name generated from the above SopsSecret in our application’s Helm config, and then commit our changes and push to git. To modify the secrets, we’d do the following

# Decrypt secrets
$ sops -d --encrypted-suffix=secret_templates path/to/app/templates/sopssecrets.yaml

# Make changes to secrets

# Encrypt secrets
$ sops -e --encrypted-suffix=secret_templates path/to/app/templates/sopssecrets.yaml

# Commit changes and push

In this workflow, ArgoCD does not need to decrypt the SopsSecret object. The sops-secrets-operator which has the necessary permission to decrypt the secret decrypts the SopsSecret objects to create Kubernetes native Secret objects.

Summary

We have been successfully using sops-secrets-operator at Zapier for a few years now.

Pros

• Intermediate CI/CD tooling does not need to have access to decrypt secrets

• Works on simple and proven technology like sops

• Secrets are encrypted and stored in git

Cons

• Duplicate secrets for each Kubernetes cluster. When secrets are updated, need to update the SopsSecret artifacts for each cluster

• There's a chance that a user might, by mistake, commit and push an unencrypted SopsSecret artifact to git

• No granular access control to secrets. Since we use the same KMS key to encrypt/decrypt all SopsSecret artifacts in the Argo configs repo, all users with access to the repo can view secrets for other applications they don't own.

Vault

In early 2021, we deployed Vault at Zapier, and we started fiddling with options to use Vault for managing secrets in Zapier. Among many other things, Vault provided us with a centralized place for managing secrets, a user-friendly UI and CLI, and granular access control. From our OIDC provider and service catalog, we have automated setting up separate secret engines for teams and giving team members necessary access to the team's secret engines.

To have a quick and simple replacement of sops-secrets-operator, we used external-secrets to manage secrets in Kubernetes. Similar to our above workflow with SopsSecret artifacts, we'd add ExternalSecret artifact(s) in the templates directory of the application's Helm chart. The external-secrets operator will reconcile these ExternalSecret objects and create native Kubernetes Secret objects by fetching the secrets from Vault. An example ExternalSecret will look something like the below.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-example
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: example-sync
  data:
  - secretKey: foobar
    remoteRef:
      key: secret/foo
      property: my-value
---
# will create a secret with:
kind: Secret
metadata:
  name: example-sync
data:
  foobar: czNjcjN0

The major win in ExternalSecret definition over SopsSecret is that you only have references to the actual secret in Vault, rather than the actual encrypted secret value. This means that if a secret gets updated, you won't need to update these ExternalSecret artifacts for all your Kubernetes clusters. You will only need to update these ExternalSecret artifacts when you add or remove secret keys.

However, the downside to our external-secret setup is that the external-secret service account has access to read secrets for all apps in Vault. To work around this shortcoming, we are exploring Vault CSI provider so that we can define access to specific Vault secret engines and paths based on Kubernetes service accounts and namespaces. Similar to Kubernetes secrets, on pod start and restart, the Secrets Store CSI driver communicates with the provider using gRPC to retrieve the secret content from the external Secrets Store specified in the SecretProviderClass custom resource. Then the volume is mounted in the pod as tmpfs and the secret contents are written to the volume.

We are finding Vault to be a quality secret management solution across various services and applications at Zapier, be it in Kubernetes or not.

Summary

Pros

• Granular access control for developers and services to view and update secrets

• Out of the box Kubernetes authentication mapping Kubernetes service accounts to Vault roles

• Central place to manage secrets in a secure way

• No duplicate secrets for each Kubernetes cluster.

• No chance for unencrypted secrets getting committed to repo by mistake. Secret artifacts are readable and contain only references to Vault paths.

Cons

• We define secrets (in Vault) and use secrets (in ExternalSecret, CSI Secret) in different places. Developers, initially, find it confusing

Conclusion

That's how and why we moved from Helm secrets to SOPS secrets operator to Vault for secrets management in Kubernetes at Zapier. We are thankful to the creators & maintainers of the awesome Open Source tools mentioned above for supporting our journey of secrets management in Kubernetes. We have grown and learned many things during this journey, and we will keep evolving our secret management practices for Kubernetes at Zapier to stay up to date with the best security practices and tools.