Secrets in Kubernetes are Annoying

It's all just base64

No Matter How You Spin It, It’s Just base64

There is a lot of sensitive data stored inside Kubernetes secrets. Database passwords, admin credentials, API Tokens, encryption keys, SSH keys, certificates, would you like me to keep going? After spending my fair share of time doing secrets management in Kubernetes, I have deduced that it is one of my least favorite parts of this wonderful piece of technology.

Why? Well, a few reasons.

The most glaring issue for me and is that secrets are just base64 encoded strings. I can do all the secret trickery with all the Kubernetes secret tools I want. At the end of the day, that secret is still just a base64 encoded string. To some, this may or may not be an issue. This all depends on your threat model but more than anything, this is an annoyance to me. There are many implementations of secrets management to mitigate some of the nuisance of having base64 encoded database connection strings stored in your cluster, but as I said, they will almost always fall short of the problem. Let's look at some examples.

SOPS

Sops allows teams to encrypt secrets and commit them to Git before being applied to the cluster (hopefully via GitOps). This is great for not storing the secrets either encoded or in plain text inside of Git, but once applied to the cluster...back to base64.

Sealed Secrets

It is not the same tool as SOPS, but pretty much does the same thing. Encrypt the secret in Git, and apply it to the cluster to be decrypted. kubectl get secrets/db-user-pass --template={{.data.password}} | base64 -d would get around this. Did I mention the encryption/decryption key is also stored in the cluster as a base64 string?

External Secrets Operator + (Your Favorite Secret Store)

This has a different implementation from the two previous methods, but as I stated, we ended up back in the same place as before. Secrets are stored in Vault for example, ESO will reach out to Vault grab the secret, and apply it in the cluster. Though we end with the same result, this method does make rotating secrets much easier, which I will get to later.

My Attack Vector/Threat Model

While researching for this post, I discovered this blog: Plain Kubernetes Secrets are fine:

The design of the Secrets API dates back to before Kubernetes v0.12. In a thread that predates the original design document, there's a line that hints at why people might be confused by Secrets:

Its hard to evaluate these alternatives without a threat model

That's exactly the issue. The naive approach to securing software is to blindly implement a checklist of security features. But a deeper understanding of security will quickly uncover that perfect security is impossible; you have to make trade-offs and prioritize the most likely scenarios. Creating a threat model can help you make those decisions.

This is a very good blog post and highlights many of the reasons why though base64 encoded strings may be an annoyance and target for many security folks’ complaints, they're sufficient once compared to the alternatives/mitigations both existing and proposed, especially with the threat model(s) the author provided. I do feel though that one threat is missing, which is the exfiltration of secrets from inside the cluster and via the API.

Mitigations

“Proper“ RBAC

Ideally restricting access to secrets should be solvable via RBAC. This is true, but keep in mind that any user or service account that has access to create a pod, deployment, etc. has permission to view any secret inside of the namespace via mounting the secret as a file or environment variable and referencing it inside of the pod. So the real solution is proper RBAC with least privilege.

Secrets Management

Secrets management tools are not as much mitigation as much as they are another layer in a "defense-in-depth" design. That being said, there is a potential play here. External secrets operator allows for rotating and syncing secrets inside of the cluster. Given a secrets management tool can also rotate credentials on the destination being authenticated to (host, database, application. etc.), though the secrets would be stored in plaintext, they would be rotated on an interval, which shrinks attack surface by lowering the time the compromised secrets are valid.

Defense in Depth

The real answer for mitigating exposing these secrets is defense in depth. Regardless of how these secrets are stored, whether that being encrypted, encoded, or even stored in outer space, a poor security strategy will result in their compromise. Layering the security around these secrets involves RBAC with the least privilege, using third-party authentication for API access, implementing mTLS, rotating secrets, preventing privileged pods from running unless required, and more.

Conclusion

If you are using local Kubernetes authentication in your cloud provider or on-prem, and cluster admin is compromised, you have much bigger problems to worry about. Much like most attacks, root access to a system is almost total compromise. These credentials should be guarded/monitored with utmost caution. Secondly, at the end of the day, the application needs a way to access the secret in plaintext. You can jump through all the hoops, secrets, environment variables, and encodings you want. The application still needs that plaintext connection string one way or another. In conclusion, though this is often thought of as bad design by the maintainers of Kubernetes, and occasionally a design choice I struggle to wrap my head around, there is no other choice when it comes to Kubernetes secrets than to live with it and build around it to the best your threat model sees fit.

TL;DR

Kubernetes secrets are all just base64 encoded strings at the end of the day so figure out a way to best layer your security around leaking these things. That or someone will kubectl get secrets/db-user-pass --template={{.data.password}} | base64 -d and you will have real problems on your hands.