Stateful KES for AI/ML Workloads
In this proof of concept (POC), we will explore the installation and management of a stateful Key Encryption Service (KES) within a Kubernetes (k8s) ecosystem. This guide facilitates cryptographic operations seamlessly, without exposing sensitive key material to consuming applications.
Implementing KES within Kubernetes in a stateful configuration ensures the persistence of encryption keys through pod lifecycle events and restarts. This setup offers an architectural resilience that is especially crucial in environments where relying on external Key Management Systems (KMS) is not an option or preferred.
When you have several million billion objects generated from AI/ML workloads such as LLMs, as several hundreds of applications access them simultaneously, you need to ensure the data encryption/decryption layer does not become the bottleneck and is as fast as possible since each object requires its own unique key. By storing these potentially several billions of keys in a stateful KES backend, even if the KMS goes offline for an extended period, keys can be saved to the KES until the KMS comes back online.
Prerequisites
Before proceeding, ensure you have:
- Kubernetes CLI (kubectl) installed and configured.
- Access to a Kubernetes cluster with appropriate permissions.
- MinIO Operator and CLI (mc) ready for deployment.
- Basic knowledge of Kubernetes and encryption concepts.
Deploy MinIO Operator
Setup Kubernetes Environment
Delete Previous Cluster
To avoid conflicts, remove any existing clusters:
kind delete clusters kind |
Create New Cluster
Set up a fresh Kubernetes cluster with the desired configuration using this kind-config.yaml file::
kind create cluster --config ~/operator/testing/kind-config.yaml |
Deploy MinIO Operator
Deploy the MinIO Operator using kubectl for managing MinIO instances in the cluster:
kubectl apply -k github.com/minio/operator/ |
Deploy MinIO Tenant
Deploy the MinIO Tenant with a lightweight configuration suitable for development and testing purposes:
kubectl apply -k github.com/minio/operator/examples/kustomization/tenant-lite |
Configure Key Encryption Service (KES)
Create Ubuntu Pod
Deploy an Ubuntu pod to host the KES service within the MinIO tenant namespace:
cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: ubuntu namespace: tenant-lite labels: app: ubuntu spec: containers: - image: ubuntu command: - "sleep" - "604800" imagePullPolicy: IfNotPresent name: ubuntu restartPolicy: Always EOF |
Install KES and mc
Execute these commands within the Ubuntu pod to install KES and MinIO Client (mc):
apt update apt install wget wget https://github.com/minio/kes/releases/latest/download/kes-linux-amd64 mv kes-linux-amd64 kes chmod +x kes mv kes /usr/local/bin/kes wget https://dl.min.io/client/mc/release/linux-amd64/mc chmod +x mc mv mc /usr/local/bin/mc |
Configure Persistent Storage for KES
Persistent Directory Creation
Create a directory within the Ubuntu pod for KES configuration and data files where KES is located:
rm -rf ~/kes mkdir ~/kes cd ~/kes touch init.yml |
Identity Management
Generate the necessary identities required for KES and MinIO authentication:
cd ~/kes kes identity new --key sys-admin.key --cert sys-admin.crt kes-sys-admin kes identity new --key minio-admin.key --cert minio-admin.crt minio-admin kes identity new --key minio.key --cert minio.crt minio kes identity new --ip "10.244.2.7" localhost # will generate private.key and public.crt | |____ IP Address of the Ubuntu Pod. |
Expected 4 identities.
root@ubuntu:/# cd ~/kes root@ubuntu:~/kes# ls data init.yml minio-admin.crt minio-admin.key minio.crt minio.key private.key public.crt sys-admin.crt sys-admin.key | | | | | | | | |__________|___ minio |________________|___ kes-sys-admin | | |_______________|___ minio-admin root@ubuntu:~/kes# |
Create a KES unseal key.
cat /dev/urandom | head -c 32 | base64 # put the result in the .bashrc vi ~/.bashrc export KES_UNSEAL_KEY=<VALUE-FROM-ABOVE-COMMAND> source ~/.bashrc echo $KES_UNSEAL_KEY # it should print the value |
Initialize KES Deployment
KES Configuration - Edit/Create KES config file
Configure the KES service by setting up the necessary configuration parameters:
cd ~/kes echo "version: v1" > ~/kes/init.yml echo "address: 0.0.0.0:7373" >> ~/kes/init.yml echo "" >> ~/kes/init.yml echo "tls:" >> ~/kes/init.yml echo " key: private.key" >> ~/kes/init.yml echo " cert: public.crt" >> ~/kes/init.yml echo " client:" >> ~/kes/init.yml echo " verify_cert: false" >> ~/kes/init.yml echo "" >> ~/kes/init.yml echo "system:" >> ~/kes/init.yml echo " admin:" >> ~/kes/init.yml echo " identity: $(kes identity of sys-admin.crt)" >> ~/kes/init.yml echo "" >> ~/kes/init.yml echo "unseal:" >> ~/kes/init.yml echo " environment:" >> ~/kes/init.yml echo " name: KES_UNSEAL_KEY" >> ~/kes/init.yml echo "" >> ~/kes/init.yml echo "enclave:" >> ~/kes/init.yml echo " default:" >> ~/kes/init.yml echo " admin:" >> ~/kes/init.yml echo " identity: $(kes identity of minio-admin.crt)" >> ~/kes/init.yml echo " policy:" >> ~/kes/init.yml echo " minio:" >> ~/kes/init.yml echo " allow:" >> ~/kes/init.yml echo " - /v1/api" >> ~/kes/init.yml echo " - /v1/log/audit" >> ~/kes/init.yml echo " - /v1/log/error" >> ~/kes/init.yml echo " - /v1/key/create/*" >> ~/kes/init.yml echo " - /v1/key/generate/*" >> ~/kes/init.yml echo " - /v1/key/decrypt/*" >> ~/kes/init.yml echo " - /v1/key/bulk/decrypt/*" >> ~/kes/init.yml |
Initialization
Initialize the KES server with the newly created configuration file:
cd ~/kes # where init.yml is saved kes init --config init.yml ~/kes/data |
Expected is
root@ubuntu:~/kes# cd ~/kes # where init.yml is saved kes init --config init.yml ~/kes/data TLS: · Private Key: private.key · Certificate: public.crt System: · Identity: 1a65f6f86c3268b30528fe4aab88fc6994730346e1c1863052fa3fa192d77d3e Unseal: · Environment: KES_UNSEAL_KEY ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Initialized KES v0.22.3 in /root/kes/data │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ root@ubuntu:~/kes# |
Start KES Server
Start the KES server and verify it's running correctly:
kes server ~/kes/data |
Expected is:
root@ubuntu:~/kes# kes server ~/kes/data Copyright MinIO, Inc. https://min.io License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html Version v0.22.3 linux/amd64 Endpoints https://127.0.0.1:7373 https://10.244.4.2:7373 mTLS skip verify Client certificates are not verified Mem Lock off Failed to lock RAM pages. Consider granting CAP_IPC_LOCK |
Configure MinIO with KES
Assign Policies
Link MinIO and KES together by assigning the appropriate policies and identities.
In Ubuntu Pod Terminal where KES is located: Assign MinIO identity to MinIO policy.
cd ~/kes export KES_SERVER=https://127.0.0.1:7373 export KES_CLIENT_KEY=minio-admin.key export KES_CLIENT_CERT=minio-admin.crt kes policy assign -k minio $(kes identity of minio.crt) |
Expected:
root@ubuntu:/# cd ~/kes export KES_SERVER=https://127.0.0.1:7373 export KES_CLIENT_KEY=minio-admin.key export KES_CLIENT_CERT=minio-admin.crt kes policy assign -k minio $(kes identity of minio.crt) root@ubuntu:~/kes# root@ubuntu:~/kes# root@ubuntu:~/kes# root@ubuntu:~/kes# root@ubuntu:~/kes# root@ubuntu:~/kes# printenv | grep KES_SERVER KES_SERVER=https://127.0.0.1:7373 root@ubuntu:~/kes# |
MinIO Server Setup
Establish the connection between MinIO and KES, defining the encryption endpoint and credentials:
- Create kes-minio secret
- Copy ~/kes/minio.key and ~/kes/minio.crt to your laptop from the ubuntu pod
These two files below come from this line: kes identity new --key minio.key --cert minio.crt minio
# /Users/aj/minio/private.key is ~/kes/minio.key # /Users/aj/minio/public.crt is ~/kes/minio.crt kubectl create secret generic kes-minio -n tenant-lite --from-file=/Users/aj/minio/private.key --from-file=/Users/aj/minio/public.crt |
- Create kes-minio-public secret:
- Copy ~/kes/private.key and ~/kes/public.crt to your laptop from the ubuntu pod
- These two files below comes from this line: kes identity new --ip "10.244.2.7" localhost:
# /Users/aj/minio/private.key is ~/kes/private.key # /Users/aj/minio/public.crt is ~/kes/public.crt kubectl create secret generic kes-minio-public -n tenant-lite --from-file=/Users/aj/minio/private.key --from-file=/Users/aj/minio/public.crt |
k edit tenant -n tenant-lite |
apiVersion: minio.min.io/v2 kind: Tenant metadata: name: storage namespace: minio-tenant spec: # externalClientCertSecrets is to share the secret with the MinIO Pods: # Under: /tmp/certs/client-0 You will find: # client.crt and client.key # And we can use these files to setup KES in k8s externalClientCertSecrets: - name: kes-minio type: Opaque - name: kes-minio-public type: Opaque env: # Set MINIO_KMS_KES_ENDPOINT # It is the IP of the Ubuntu Pod. - name: MINIO_KMS_KES_ENDPOINT value: "https://<IP-ADDRESS-OF-UBUNTU-POD>:7373" # Set MinIO Client Credentials, it comes from kes-minio secret - name: MINIO_KMS_KES_CERT_FILE value: "/tmp/certs/client-0/client.crt" # Set MinIO Client Credentials, it comes from kes-minio secret - name: MINIO_KMS_KES_KEY_FILE value: "/tmp/certs/client-0/client.key" # Set MinIO Default Key - name: MINIO_KMS_KES_KEY_NAME value: "minio-default-key" # Trust the KES Server Certificate, it comes from kes-minio-public secret - name: MINIO_KMS_KES_CAPATH value: "/tmp/certs/client-1/client.crt" # Root User - name: MINIO_ROOT_USER value: minio # ROOT Password: - name: MINIO_ROOT_PASSWORD value: minio123 |
Encryption Operations
Conduct encryption operations by creating an encrypted bucket and configuring server-side encryption:
mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123 mc rb myminio/my-bucket --force # remove previous bucket to start fresh mc mb myminio/my-bucket # create new bucket mc admin kms key create myminio minio-my-bucket # create key mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket |
Verification and Testing of the Results
Ensure that your setup is correct by verifying that keys can be created and accessed through MinIO:
root@ubuntu:/# mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123 Added `myminio` successfully. root@ubuntu:/# mc rb myminio/my-bucket --force # remove previous bucket to start fresh mc: <ERROR> Unable to validate target `myminio/my-bucket`. Bucket `my-bucket` does not exist. root@ubuntu:/# mc mb myminio/my-bucket # create new bucket Bucket created successfully `myminio/my-bucket`. root@ubuntu:/# mc admin kms key create myminio minio-my-bucket # create key Created master key `minio-my-bucket` successfully root@ubuntu:/# mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket Auto encryption configuration has been set successfully for myminio/my-bucket root@ubuntu:/# |
Final Thoughts
After following the outlined steps, you have successfully deployed a stateful KES in your Kubernetes cluster. This configuration allows you to manage encryption keys within your cluster, simplifying your deployment and reducing dependencies on external KMS solutions. This ensures your AI/ML workloads perform smoothly without interruption that depend on this critical feature in the infrastructure.
The integration of stateful KES in Kubernetes with this hands-on experience reinforces your skills as a DevOps engineer as you continue to explore. If you have any questions on deploying stateful KES in your Kubernetes cluster be sure to reach out to us on Slack!