How to Secure Kubernetes Secrets

Kubernetes has become the most popular container orchestrator in the industry. It’s powerful, scalable, and flexible enough to deploy on-premises or in the cloud. However, with these benefits come some security concerns that are worth addressing. Kubernetes Secrets help you manage an application’s sensitive data across your cluster, reducing the potential risks of managing non-centralized secrets.

Securing access to cluster secrets is critical because unauthorised access to private data violates personal privacy and can lead to compliance violations. 

This article talks about Kubernetes secrets, why secrets are important, how you can make secrets secure, and the best practices.

What are K8s Secrets?

Credentials or secrets required by applications. Kubernetes secrets are particular objects that can store and manage sensitive data such as authentication tokens, passwords, SSH keys, or cloud access keys in your clusters.

Secrets in Kubernetes are stored in base64 encoded format. Kubernetes stores secrets in plain text format unless you’ve specified an encryption provider. A secret in Kubernetes can consume using environmental variables or files in a volume.

Secrets are a powerful tool in your Kubernetes toolkit. One of the most significant advantages of using secrets is that the cluster itself manages secrets. You don’t have to manage them yourself or store them elsewhere — they’re just there for you when you need them. As a result, there’s no tricky setup process, and there isn’t any extra work will involve in creating and storing secrets: they live right alongside other Kubernetes objects like pods and services.

How Secure are your K8s Secrets?

It is an awful practice to store sensitive data, such as passwords, authentication tokens, and SSH keys, in plaintext on a container. However, containers need this data to perform basic operations like integrating with other systems. To this end, Kubernetes provides an object called Secret, which you can use to store sensitive data. Below is an example of a secret.

apiVersion: v1
data:
super-secret: aHR0cDovL2pvYnMuc29sdXRvLmNvbS9hcHBseS95MDVYSXEvUHJvZHVjdGlvbi1FbmdpbmVlci1EZXZPcHM=
kind: Secret
metadata:
username: admin
password: mUV2Sw7eLbfp
type: Opaque

Kubernetes offers network security resources such as controlling which containers and pods communicate over the network and even some basic secret management capabilities. However, it does not help secure secrets and only provides limited functionality.

One downside to Kubernetes secrets is that they will not encrypt by default, thus exposing your system to potential security risks if someone gains access to the unencrypted data. In addition, secrets can leak in several ways: source code management tools like Git, application’s source code, logs, etc. Hence, secrets will always store using proper encryption techniques.

You should not consider your data secure just by preserving the sensitive data in a secret object. Kubernetes has no robust mechanism to share, encrypt and manage secrets across the clusters. Therefore, you must avoid preserving sensitive data such as passwords, SSH keys, and authentication tokens as plain text.

How Do Kubernetes Secrets Work?

There are two types of secrets in Kubernetes:

• Built-in secrets—Kubernetes Service Accounts automatically create secrets and attach them to containers with API Credentials. This mechanism will disable or override if it raises security concerns.
• Custom secrets—you can define your sensitive data and create a secret to store it.

There are two primary ways of creating Kubernetes secrets:

• Automatically—using kubectl create secret—specify one or more files that include sensitive data, and these files are then packaged as a secret.

kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt

• Manually—create a secret in a JSON or YAML file, then write the code of the object and create a secret from it using kubectl apply

kubectl get secret mysecret -o yaml

To decode a secret from Base64 encoding to plaintext, use kubectl get secret to view the content of the secret. Then decode the sensitive data like this:

echo 'MWYyZDFlMmU2N2Rm’ | base64 –decode

Kubernetes Cluster Secrets Challenges

Kubernetes service accounts can automatically create built-in secrets and bind them with the containers. Note that they may deactivate or override. Customised Secrets allow you to enter sensitive data and create a secret to keep it. 

Kubernetes secret deployments are more secure than direct deployment in the Docker image. However, Kubernetes Secrets management also has its limitations. Kubernetes Secrets keeps passwords and usernames by encoding them in base-64 format. Keep in mind that text encoding is never secure. Furthermore, secrets are only accessible inside the cluster in which they are stored.

Secrets in Kubernetes are preserved in etcd, a high-performance, distributed key/value storage system. However, it lacks several critical features necessary for storing sensitive data, such as an audit log, insights into crucial age, and automated key rotation. However, etcd lacks essential security features such as automatic key rotation, audit log, etc. Moreover, since the secrets are stored etcd unencrypted, anyone having access to the cluster can access those secrets.

Built-in Security Controls in Kubernetes

Kubernetes implements several built-in security controls to provide a baseline of security for secrets:

• One pod cannot access the secrets of another.
• A secret is made available to a node if a pod will schedule on that node and needs it.
• The secret is saved in “tmpfs” so it is not written to persistent storage. This file system is RAM-backed and will not survive a reboot.
• The maximum permissible size of secrets can only be 1 MB. This helps protect the API server memory resources kubelet from abuse.
• The local copy of the secret will also delete once the pod containing the secret is removed.
• When there are many containers in a Pod, each container must explicitly request the secret volume to be visible inside the container.

Although these controls are essential, they are by no means adequate for production-grade implementations. As a result, Kubernetes administrators must have extra safeguards, mainly when dealing with systems not part of the Kubernetes ecosystem.

Problems with the built-in Secrets Mechanism in Kubernetes

No zero-trust system – It would help if you decrypted secrets only when necessary. Kubernetes, on the other hand, does not natively offer this capability. As a result, any user with root access to a Kubernetes node may be able to see the secrets inside a node. Moreover, in a zero-trust system, once you’ve encrypted a secret, there is no way in which you can decrypt it.

Albeit providing support for role-based access, the existing permissions for Kubernetes secrets get and set. Therefore, although the secrets are encrypted at rest, there is no way to enable the user to get a secret encrypted. Hence a zero-trust system is imperative.

Insecure Storage of Secrets – In Kubernetes, secrets are stored inside a data file known as “etcd.” Since the contents of “etcd” are not encrypted by default; the administrators must limit their access privileges appropriately. Additionally, Kubernetes secrets will not encrypt by default and will store as base64 strings. 

So you cannot commit the files that contain these secrets into the source control as-is since anyone with access to the repository can also get access to the secrets. Secrets can be exposed through application code, logs, debug records, or even source code management tools such as Gitlab and Github.

External Risks – Kubernetes secrets management and how other applications use your secrets also matters. Ensure that you don’t expose your sensitive data using tools and platforms with no security measures.

Kubernetes Secrets Management Best Practices

Securing Kubernetes is vital for maintaining a safe environment where your data can only be accessed when needed. Here are some ways to protect your data in Kubernetes environments with secret management best practices.

Use a Centralized Secret Management Solution

A centralised secrets management solution such as Akeyless or Conjur simplifies audit, access control, and secrets management by providing enterprises with a consolidated view of their complete Kubernetes security environment. It gives organisations a centralised view of all aspects of Kubernetes security. This makes it possible to control and understand how containers and services interact.

Secret Management in a Container Environment

It is recommended to ensure that secrets are only available to the containers that need them. In addition, you must ensure that secrets in a container will store securely. For example, it would help if you did not keep secrets in the disk; instead, secrets should be accessible only to the container that needs them, stored in the memory, and disappear as soon as the container shuts down.

Configuration Best Practices

Almost all data in the Kubernetes API, including secrets, will keep in a distributed data store. Setup the Kubernetes cluster configuration to encrypt the data at rest to encrypt. It would be best to use the EncryptionConfiguration object at the cluster level to allow encryption at rest and decrease the number of keys that must be managed.

Networking Best Practices

You can apply the principle of least privilege to your network traffic so that sensitive cluster components can access the services that need direct access. Also, you can limit service-to-service traffic by using the traffic filtering mechanisms of your cloud provider. Finally, you can also restrict traffic between application components using NetworkPolicy tools.

Security Best Practices

Some of the best practices in securing your Kubernetes secrets include using a centralised secrets management solution, role-based access control (RBAC), regular auditing, etc.

Secure internal communication in a Kubernetes Cluster by enabling TLS/SSL between the pods and Etcd.

• It would help if you encrypted the secret keys so that anyone who tries to steal your secret will be unable to see it. However, to access the secret keys, you need to decrypt them using a private key. 
• You should harden the system by ensuring that the secrets stored etcd, are encrypted before being written. Additionally, you should have strong key management for your symmetric encryption keys to rotate keys often. Finally, access to etcd should be restricted – only the API server should be allowed to access the etcd.
• There should be a certificate-based mutual authentication in place. It is recommended that you use a different certificate authority (CA) for safeguarding access, etcd, than the one you use for protecting access to Kubernetes.
• Since secrets inside a cluster might expose to all cluster users, you must exercise caution while uploading personal credentials, mainly if you operate in a shared workplace. In addition, it would help if you avoided including environmental variables in your logging statements since this might disclose the secret in plaintext inside the logs.
• You must enable Encryption-at-rest – this is a feature that encrypts the data contained inside.”etcd.” Since this feature is not enabled by default, you should allow it to encrypt the data saved explicitly.”etcd.“
• Kubernetes leverages Etcd to store cluster data. Incidentally, “etcd” is a distributed, highly available, and consistent key-value store. This data may contain sensitive information regarding access to deployed services. So, you need to restrict access to.”etcd” by giving access only to the admin users.

Conclusion

Kubernetes Secrets is a mechanism for managing and accessing an application’s sensitive data across the cluster. However, they are accessible to unauthorised users if you don’t follow the recommended best practices to secure them. Therefore, you should enable “Encryption at Rest”, restrict access to “etcd”, and follow other best practices outlined earlier in this article to secure your Kubernetes secrets.

Offensive360 allows organisations to create, manage and deploy custom governance policies using policy-as code. It also provides a powerful policy enforcement engine. Offensive360 codifies your policies to maximise cloud infrastructure security. The Offensive360 offers many features, including customisable policies, compliance checks, and automatic remediation.