Stateful KES for AI/ML Workloads

In this proof of concept (POC), we will explore the installation and management of a stateful Key Encryption Service (KES) within a Kubernetes (k8s) ecosystem. This guide facilitates cryptographic operations seamlessly, without exposing sensitive key material to consuming applications.

Implementing KES within Kubernetes in a stateful configuration ensures the persistence of encryption keys through pod lifecycle events and restarts. This setup offers an architectural resilience that is especially crucial in environments where relying on external Key Management Systems (KMS) is not an option or preferred.

When you have several million billion objects generated from AI/ML workloads such as LLMs, as several hundreds of applications access them simultaneously, you need to ensure the data encryption/decryption layer does not become the bottleneck and is as fast as possible since each object requires its own unique key. By storing these potentially several billions of keys in a stateful KES backend, even if the KMS goes offline for an extended period, keys can be saved to the KES until the KMS comes back online.

Prerequisites

Before proceeding, ensure you have:

Kubernetes CLI (kubectl) installed and configured.
Access to a Kubernetes cluster with appropriate permissions.
MinIO Operator and CLI (mc) ready for deployment.
Basic knowledge of Kubernetes and encryption concepts.

Deploy MinIO Operator

Setup Kubernetes Environment

Delete Previous Cluster

To avoid conflicts, remove any existing clusters:

kind delete clusters kind

Create New Cluster

Set up a fresh Kubernetes cluster with the desired configuration using this kind-config.yaml file::

kind create cluster --config ~/operator/testing/kind-config.yaml

Deploy MinIO Operator

Deploy the MinIO Operator using kubectl for managing MinIO instances in the cluster:

kubectl apply -k github.com/minio/operator/

Deploy MinIO Tenant

Deploy the MinIO Tenant with a lightweight configuration suitable for development and testing purposes:

kubectl apply -k github.com/minio/operator/examples/kustomization/tenant-lite

Configure Key Encryption Service (KES)

Create Ubuntu Pod

Deploy an Ubuntu pod to host the KES service within the MinIO tenant namespace:

cat <<EOF | kubectl apply -f -

apiVersion: v1

kind: Pod

metadata:

namespace: tenant-lite

labels:

app: ubuntu

spec:

containers:

- image: ubuntu

command:

- "sleep"

- "604800"

imagePullPolicy: IfNotPresent

restartPolicy: Always

EOF

Install KES and mc

Execute these commands within the Ubuntu pod to install KES and MinIO Client (mc):

apt update

apt install wget

wget https://github.com/minio/kes/releases/latest/download/kes-linux-amd64

mv kes-linux-amd64 kes

chmod +x kes

mv kes /usr/local/bin/kes

wget https://dl.min.io/client/mc/release/linux-amd64/mc

chmod +x mc

mv mc /usr/local/bin/mc

Configure Persistent Storage for KES

Persistent Directory Creation

Create a directory within the Ubuntu pod for KES configuration and data files where KES is located:

rm -rf ~/kes

mkdir ~/kes

cd ~/kes

touch init.yml

Identity Management

Generate the necessary identities required for KES and MinIO authentication:

cd ~/kes

kes identity new --key sys-admin.key --cert sys-admin.crt kes-sys-admin

kes identity new --key minio-admin.key --cert minio-admin.crt minio-admin

kes identity new --key minio.key --cert minio.crt minio

kes identity new --ip "10.244.2.7" localhost # will generate private.key and public.crt

|____ IP Address of the Ubuntu Pod.

Expected 4 identities.

root@ubuntu:/# cd ~/kes

root@ubuntu:~/kes# ls

data init.yml minio-admin.crt minio-admin.key minio.crt minio.key private.key public.crt sys-admin.crt sys-admin.key

| | | | | |

| | |__________|___ minio |________________|___ kes-sys-admin

| |

|_______________|___ minio-admin

root@ubuntu:~/kes#

Create a KES unseal key.

cat /dev/urandom | head -c 32 | base64 # put the result in the .bashrc

vi ~/.bashrc

export KES_UNSEAL_KEY=<VALUE-FROM-ABOVE-COMMAND>

source ~/.bashrc

echo $KES_UNSEAL_KEY # it should print the value

Initialize KES Deployment

KES Configuration - Edit/Create KES config file

Configure the KES service by setting up the necessary configuration parameters:

cd ~/kes

echo "version: v1" > ~/kes/init.yml

echo "address: 0.0.0.0:7373" >> ~/kes/init.yml

echo "" >> ~/kes/init.yml

echo "tls:" >> ~/kes/init.yml

echo " key: private.key" >> ~/kes/init.yml

echo " cert: public.crt" >> ~/kes/init.yml

echo " client:" >> ~/kes/init.yml

echo " verify_cert: false" >> ~/kes/init.yml

echo "" >> ~/kes/init.yml

echo "system:" >> ~/kes/init.yml

echo " admin:" >> ~/kes/init.yml

echo " identity: $(kes identity of sys-admin.crt)" >> ~/kes/init.yml

echo "" >> ~/kes/init.yml

echo "unseal:" >> ~/kes/init.yml

echo " environment:" >> ~/kes/init.yml

echo " name: KES_UNSEAL_KEY" >> ~/kes/init.yml

echo "" >> ~/kes/init.yml

echo "enclave:" >> ~/kes/init.yml

echo " default:" >> ~/kes/init.yml

echo " admin:" >> ~/kes/init.yml

echo " identity: $(kes identity of minio-admin.crt)" >> ~/kes/init.yml

echo " policy:" >> ~/kes/init.yml

echo " minio:" >> ~/kes/init.yml

echo " allow:" >> ~/kes/init.yml

echo " - /v1/api" >> ~/kes/init.yml

echo " - /v1/log/audit" >> ~/kes/init.yml

echo " - /v1/log/error" >> ~/kes/init.yml

echo " - /v1/key/create/*" >> ~/kes/init.yml

echo " - /v1/key/generate/*" >> ~/kes/init.yml

echo " - /v1/key/decrypt/*" >> ~/kes/init.yml

echo " - /v1/key/bulk/decrypt/*" >> ~/kes/init.yml

Initialization

Initialize the KES server with the newly created configuration file:

cd ~/kes # where init.yml is saved

kes init --config init.yml ~/kes/data

Expected is

root@ubuntu:~/kes# cd ~/kes # where init.yml is saved

kes init --config init.yml ~/kes/data

TLS:

· Private Key: private.key

· Certificate: public.crt

System:

· Identity: 1a65f6f86c3268b30528fe4aab88fc6994730346e1c1863052fa3fa192d77d3e

Unseal:

· Environment: KES_UNSEAL_KEY

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮

│ Initialized KES v0.22.3 in /root/kes/data │

╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

root@ubuntu:~/kes#

Start KES Server

Start the KES server and verify it's running correctly:

kes server ~/kes/data

Expected is:

root@ubuntu:~/kes# kes server ~/kes/data

License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

Version v0.22.3 linux/amd64

Endpoints https://127.0.0.1:7373

https://10.244.4.2:7373

mTLS skip verify Client certificates are not verified

Mem Lock off Failed to lock RAM pages. Consider granting CAP_IPC_LOCK

Configure MinIO with KES

Assign Policies

Link MinIO and KES together by assigning the appropriate policies and identities.

In Ubuntu Pod Terminal where KES is located: Assign MinIO identity to MinIO policy.

cd ~/kes

export KES_SERVER=https://127.0.0.1:7373

export KES_CLIENT_KEY=minio-admin.key

export KES_CLIENT_CERT=minio-admin.crt

kes policy assign -k minio $(kes identity of minio.crt)

Expected:

root@ubuntu:/# cd ~/kes

export KES_SERVER=https://127.0.0.1:7373

export KES_CLIENT_KEY=minio-admin.key

export KES_CLIENT_CERT=minio-admin.crt

kes policy assign -k minio $(kes identity of minio.crt)

root@ubuntu:~/kes#

root@ubuntu:~/kes# printenv | grep KES_SERVER

KES_SERVER=https://127.0.0.1:7373

root@ubuntu:~/kes#

MinIO Server Setup

Establish the connection between MinIO and KES, defining the encryption endpoint and credentials:

Create kes-minio secret
Copy ~/kes/minio.key and ~/kes/minio.crt to your laptop from the ubuntu pod

These two files below come from this line: kes identity new --key minio.key --cert minio.crt minio

# /Users/aj/minio/private.key is ~/kes/minio.key

# /Users/aj/minio/public.crt is ~/kes/minio.crt

kubectl create secret generic kes-minio -n tenant-lite --from-file=/Users/aj/minio/private.key --from-file=/Users/aj/minio/public.crt

Create kes-minio-public secret:
- Copy ~/kes/private.key and ~/kes/public.crt to your laptop from the ubuntu pod
- These two files below comes from this line: kes identity new --ip "10.244.2.7" localhost:

# /Users/aj/minio/private.key is ~/kes/private.key

# /Users/aj/minio/public.crt is ~/kes/public.crt

kubectl create secret generic kes-minio-public -n tenant-lite --from-file=/Users/aj/minio/private.key --from-file=/Users/aj/minio/public.crt

k edit tenant -n tenant-lite

apiVersion: minio.min.io/v2

kind: Tenant

metadata:

namespace: minio-tenant

spec:

# externalClientCertSecrets is to share the secret with the MinIO Pods:

# Under: /tmp/certs/client-0 You will find:

# client.crt and client.key

# And we can use these files to setup KES in k8s

externalClientCertSecrets:

- name: kes-minio

type: Opaque

- name: kes-minio-public

type: Opaque

env:

# Set MINIO_KMS_KES_ENDPOINT

# It is the IP of the Ubuntu Pod.

- name: MINIO_KMS_KES_ENDPOINT

value: "https://<IP-ADDRESS-OF-UBUNTU-POD>:7373"

# Set MinIO Client Credentials, it comes from kes-minio secret

- name: MINIO_KMS_KES_CERT_FILE

value: "/tmp/certs/client-0/client.crt"

# Set MinIO Client Credentials, it comes from kes-minio secret

- name: MINIO_KMS_KES_KEY_FILE

value: "/tmp/certs/client-0/client.key"

# Set MinIO Default Key

- name: MINIO_KMS_KES_KEY_NAME

value: "minio-default-key"

# Trust the KES Server Certificate, it comes from kes-minio-public secret

- name: MINIO_KMS_KES_CAPATH

value: "/tmp/certs/client-1/client.crt"

# Root User

- name: MINIO_ROOT_USER

value: minio

# ROOT Password:

- name: MINIO_ROOT_PASSWORD

value: minio123

Encryption Operations

Conduct encryption operations by creating an encrypted bucket and configuring server-side encryption:

mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123

mc rb myminio/my-bucket --force # remove previous bucket to start fresh

mc mb myminio/my-bucket # create new bucket

mc admin kms key create myminio minio-my-bucket # create key

mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket

Verification and Testing of the Results

Ensure that your setup is correct by verifying that keys can be created and accessed through MinIO:

root@ubuntu:/# mc alias set myminio https://minio.tenant-lite.svc.cluster.local:443 minio minio123

Added `myminio` successfully.

root@ubuntu:/# mc rb myminio/my-bucket --force # remove previous bucket to start fresh

mc: <ERROR> Unable to validate target `myminio/my-bucket`. Bucket `my-bucket` does not exist.

root@ubuntu:/# mc mb myminio/my-bucket # create new bucket

Bucket created successfully `myminio/my-bucket`.

root@ubuntu:/# mc admin kms key create myminio minio-my-bucket # create key

Created master key `minio-my-bucket` successfully

root@ubuntu:/# mc encrypt set sse-kms minio-my-bucket myminio/my-bucket # encrypt bucket

Auto encryption configuration has been set successfully for myminio/my-bucket

root@ubuntu:/#

Final Thoughts

After following the outlined steps, you have successfully deployed a stateful KES in your Kubernetes cluster. This configuration allows you to manage encryption keys within your cluster, simplifying your deployment and reducing dependencies on external KMS solutions. This ensures your AI/ML workloads perform smoothly without interruption that depend on this critical feature in the infrastructure.

The integration of stateful KES in Kubernetes with this hands-on experience reinforces your skills as a DevOps engineer as you continue to explore. If you have any questions on deploying stateful KES in your Kubernetes cluster be sure to reach out to us on Slack!