Kubernetes has become the most popular container orchestrator in the industry. It’s powerful, scalable, and flexible enough to deploy on-premises or in the cloud. However, with these benefits come some security concerns that are worth addressing. Kubernetes Secrets help you manage an application’s sensitive data across your cluster, reducing the potential risks of managing non-centralized secrets.
Securing access to cluster secrets is critical because unauthorised access to private data violates personal privacy and can lead to compliance violations.
This article talks about Kubernetes secrets, why secrets are important, how you can make secrets secure, and the best practices.
Credentials or secrets required by applications. Kubernetes secrets are particular objects that can store and manage sensitive data such as authentication tokens, passwords, SSH keys, or cloud access keys in your clusters.
Secrets in Kubernetes are stored in base64 encoded format. Kubernetes stores secrets in plain text format unless you’ve specified an encryption provider. A secret in Kubernetes can consume using environmental variables or files in a volume.
Secrets are a powerful tool in your Kubernetes toolkit. One of the most significant advantages of using secrets is that the cluster itself manages secrets. You don’t have to manage them yourself or store them elsewhere — they’re just there for you when you need them. As a result, there’s no tricky setup process, and there isn’t any extra work will involve in creating and storing secrets: they live right alongside other Kubernetes objects like pods and services.
It is an awful practice to store sensitive data, such as passwords, authentication tokens, and SSH keys, in plaintext on a container. However, containers need this data to perform basic operations like integrating with other systems. To this end, Kubernetes provides an object called Secret, which you can use to store sensitive data. Below is an example of a secret.
apiVersion: v1
data:
super-secret: aHR0cDovL2pvYnMuc29sdXRvLmNvbS9hcHBseS95MDVYSXEvUHJvZHVjdGlvbi1FbmdpbmVlci1EZXZPcHM=
kind: Secret
metadata:
username: admin
password: mUV2Sw7eLbfp
type: Opaque
Kubernetes offers network security resources such as controlling which containers and pods communicate over the network and even some basic secret management capabilities. However, it does not help secure secrets and only provides limited functionality.
etcd
”One downside to Kubernetes secrets is that they will not encrypt by default, thus exposing your system to potential security risks if someone gains access to the unencrypted data. In addition, secrets can leak in several ways: source code management tools like Git, application’s source code, logs, etc. Hence, secrets will always store using proper encryption techniques.
You should not consider your data secure just by preserving the sensitive data in a secret object. Kubernetes has no robust mechanism to share, encrypt and manage secrets across the clusters. Therefore, you must avoid preserving sensitive data such as passwords, SSH keys, and authentication tokens as plain text.
There are two types of secrets in Kubernetes:
There are two primary ways of creating Kubernetes secrets:
kubectl create secret
—specify one or more files that include sensitive data, and these files are then packaged as a secret.kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt
kubectl apply
kubectl get secret mysecret -o yaml
To decode a secret from Base64 encoding to plaintext, use kubectl
get secret to view the content of the secret. Then decode the sensitive data like this:
echo 'MWYyZDFlM
mU2N2Rm’ | base64 –decode
Kubernetes service accounts can automatically create built-in secrets and bind them with the containers. Note that they may deactivate or override. Customised Secrets allow you to enter sensitive data and create a secret to keep it.
Kubernetes secret deployments are more secure than direct deployment in the Docker image. However, Kubernetes Secrets management also has its limitations. Kubernetes Secrets keeps passwords and usernames by encoding them in base-64 format. Keep in mind that text encoding is never secure. Furthermore, secrets are only accessible inside the cluster in which they are stored.
Secrets in Kubernetes are preserved in etcd, a high-performance, distributed key/value storage system. However, it lacks several critical features necessary for storing sensitive data, such as an audit log, insights into crucial age, and automated key rotation. However, etcd lacks essential security features such as automatic key rotation, audit log, etc. Moreover, since the secrets are stored etcd
unencrypted, anyone having access to the cluster can access those secrets.
Kubernetes implements several built-in security controls to provide a baseline of security for secrets:
tmpfs
” so it is not written to persistent storage. This file system is RAM-backed and will not survive a reboot.kubelet
from abuse.Although these controls are essential, they are by no means adequate for production-grade implementations. As a result, Kubernetes administrators must have extra safeguards, mainly when dealing with systems not part of the Kubernetes ecosystem.
No zero-trust system – It would help if you decrypted secrets only when necessary. Kubernetes, on the other hand, does not natively offer this capability. As a result, any user with root access to a Kubernetes node may be able to see the secrets inside a node. Moreover, in a zero-trust system, once you’ve encrypted a secret, there is no way in which you can decrypt it.
Albeit providing support for role-based access, the existing permissions for Kubernetes secrets get and set. Therefore, although the secrets are encrypted at rest, there is no way to enable the user to get a secret encrypted. Hence a zero-trust system is imperative.
Insecure Storage of Secrets – In Kubernetes, secrets are stored inside a data file known as “etcd
.” Since the contents of “etcd
” are not encrypted by default; the administrators must limit their access privileges appropriately. Additionally, Kubernetes secrets will not encrypt by default and will store as base64 strings.
So you cannot commit the files that contain these secrets into the source control as-is since anyone with access to the repository can also get access to the secrets. Secrets can be exposed through application code, logs, debug records, or even source code management tools such as Gitlab and Github.
External Risks – Kubernetes secrets management and how other applications use your secrets also matters. Ensure that you don’t expose your sensitive data using tools and platforms with no security measures.
Securing Kubernetes is vital for maintaining a safe environment where your data can only be accessed when needed. Here are some ways to protect your data in Kubernetes environments with secret management best practices.
A centralised secrets management solution such as Akeyless or Conjur simplifies audit, access control, and secrets management by providing enterprises with a consolidated view of their complete Kubernetes security environment. It gives organisations a centralised view of all aspects of Kubernetes security. This makes it possible to control and understand how containers and services interact.
It is recommended to ensure that secrets are only available to the containers that need them. In addition, you must ensure that secrets in a container will store securely. For example, it would help if you did not keep secrets in the disk; instead, secrets should be accessible only to the container that needs them, stored in the memory, and disappear as soon as the container shuts down.
Almost all data in the Kubernetes API, including secrets, will keep in a distributed data store. Setup the Kubernetes cluster configuration to encrypt the data at rest to encrypt. It would be best to use the EncryptionConfiguration object at the cluster level to allow encryption at rest and decrease the number of keys that must be managed.
You can apply the principle of least privilege to your network traffic so that sensitive cluster components can access the services that need direct access. Also, you can limit service-to-service traffic by using the traffic filtering mechanisms of your cloud provider. Finally, you can also restrict traffic between application components using NetworkPolicy tools.
Some of the best practices in securing your Kubernetes secrets include using a centralised secrets management solution, role-based access control (RBAC), regular auditing, etc.
Secure internal communication in a Kubernetes Cluster by enabling TLS/SSL between the pods and Etcd.
etcd.
“etcd
” is a distributed, highly available, and consistent key-value store. This data may contain sensitive information regarding access to deployed services. So, you need to restrict access to.”etcd
” by giving access only to the admin users.Kubernetes Secrets is a mechanism for managing and accessing an application’s sensitive data across the cluster. However, they are accessible to unauthorised users if you don’t follow the recommended best practices to secure them. Therefore, you should enable “Encryption at Rest”, restrict access to “etcd”, and follow other best practices outlined earlier in this article to secure your Kubernetes secrets.
Offensive360 allows organisations to create, manage and deploy custom governance policies using policy-as code. It also provides a powerful policy enforcement engine. Offensive360 codifies your policies to maximise cloud infrastructure security. The Offensive360 offers many features, including customisable policies, compliance checks, and automatic remediation.