MinIO Operator with KES backed by Vault

AJ AJ on Security 5 March 2024

In order to provide functionality for regulatory compliance around secure locking and erasure, MinIO encrypts objects at the storage layer by using Server-Side Encryption (SSE) to protect objects as part of write operations. MinIO does this with extreme efficiency – benchmarks show that MinIO is capable of encrypting/decrypting at close to wire speed.

The secret sauce MinIO uses is Single Instruction Multiple Data (SIMD). Generally, you can only send one CPU instruction at a time and wait for a response before sending the next instruction. This is highly inefficient, especially when performing thousands – if not millions – of encryption and decryption instructions per second. MinIO takes advantage of SIMD so it can send multiple instructions in a single request to be processed by the CPU. We’ve written this in assembly within GoLang so we are as close to the hardware layer as possible to take full advantage of the typically underutilized CPU power to perform cryptographic operations at scale.

In a previous post we showed you how to get started with MinIO and Vault in a bare-metal environment using KES. In this post we’ll show you how to configure MinIO Operator with KES (Keys Encryption System) and Vault in a cloud native way in Kubernetes. This will enable you to automate the process as you scale and use Kubernetes resources to configure them.

Prerequisites

Before we begin, ensure that you have the following prerequisites in place:

A Kubernetes cluster (for this tutorial, we will use kind to create a local cluster)
kubectl command-line tool installed on your local machine
git for cloning the necessary repositories

Step 1: Deploy a Kubernetes Cluster with kind

To get started, we will create a Kubernetes cluster using kind. This will provide us with a local environment to deploy and test our MinIO setup.

First, create a kind configuration file named kind-config.yaml with the following command:

$ cat > kind-config.yaml <<EOF

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

- role: worker

EOF

$ kind create cluster --config kind-config.yml

Step 2: Deploy MinIO Operator

With our Kubernetes cluster up and running, we can now deploy the MinIO Operator using the kustomization plugin. The MinIO Operator simplifies the deployment and management of MinIO instances in a Kubernetes environment.

Execute the following command to deploy the MinIO Operator in kind cluster:

$ kubectl apply -k github.com/minio/operator

Wait for all the pods to come online in the minio-operator namespace before proceeding to the next step.

Step 3: Set up HashiCorp Vault

HashiCorp Vault is a powerful secrets management tool that we will use to securely store and manage our encryption keys. To set up Vault in our Kubernetes cluster, we will use the kubernetes-vault repository.

Clone the repository by running the following command:

$ git clone https://github.com/scriptcamp/kubernetes-vault.git

$ cd kubernetes-vault/vault-manifests

Next, set up the necessary RBAC (Role-Based Access Control) rules, create Vault config maps, deploy Vault services, and set up a stateful set:

$ kubectl apply -f rbac.yml

Create Vault config maps

$ kubectl apply -f configmap.yaml

Deploy Vault services

$ kubectl apply -f services.yaml

As Vault is stateful service, we need to set up a stateful set for the same. Execute the below command for the same

$ kubectl apply -f statefulset.yaml

Once the Vault setup is complete, unseal and initialize Vault.

$ kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > keys.json

$ VAULT_UNSEAL_KEY=$(cat keys.json | jq -r ".unseal_keys_b64[]")

$ echo $VAULT_UNSEAL_KEY

$ VAULT_ROOT_KEY=$(cat keys.json | jq -r ".root_token")

$ echo $VAULT_ROOT_KEY

$ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

Now, enter the vault-0 POS to perform additional configuration:

$ kubectl exec -it vault-0 -- /bin/sh

Inside the pod, enable the K/V backend, create a policy, enable AppRole authentication, create a KES role, and generate an app-role ID and secret:

$ vault secrets enable -version=1 kv

Create policy

$ cat > kes-policy.hcl <<EOF

path "kv/*" {

capabilities = [ "create", "read", "delete" ]

}

$ vault policy write kes-policy kes-policy.hcl

Enable AppRole authentication

$ vault auth enable approle

Create KES role and attach policy to it

$ vault write auth/approle/role/kes-server token_num_uses=0 secret_id_num_uses=0 period=5m

$ vault write auth/approle/role/kes-server policies=kes-policy

Generate app-role ID and secret, and take a note of values

$ vault read auth/approle/role/kes-server/role-id

Key Value

--- -----

role_id b484633b-8965-08dd-0e24-d6973e6be2d2

$ vault write -f auth/approle/role/kes-server/secret-id

Key Value

--- -----

secret_id 7fd44d44-a3f1-a013-1f40-e1952496c416

secret_id_accessor b5ee0c97-5ffc-b9a7-fe0b-763757fe1033

secret_id_ttl 0s

Take note of the generated app-role ID and secret, as we will need them in the next step.

Step 4: Deploy KES using `kustomization` plugin

To deploy KES (Key Encryption Service) using the kustomization plugin, we first need to clone the MinIO repository:

$ git clone https://github.com/minio/operator.git

$ cd operator

Update the examples/kustomization/tenant-kes-encryption/kes-configuration-secret.yaml file with the appropriate KES values, including the Vault endpoint, namespace, prefix, and the generated app-role ID and secret.

keystore:

## KES configured with fs (File System mode) doesnt work in Kubernetes environments and it's not recommended

## use a real KMS

# fs:

# path: "./keys" # Path to directory. Keys will be stored as files. Not Recommended for Production.

vault:

endpoint: "http://vault.default.svc.cluster.local:8200" # The Vault endpoint

namespace: "default" # An optional Vault namespace. See: https://www.vaultproject.io/docs/enterprise/namespaces/index.html

prefix: "my-minio" # An optional K/V prefix. The server will store keys under this prefix.

approle: # AppRole credentials. See: https://www.vaultproject.io/docs/auth/approle.html

id: "b484633b-8965-08dd-0e24-d6973e6be2d2" # Your AppRole Role ID

secret: "7fd44d44-a3f1-a013-1f40-e1952496c416" # Your AppRole Secret ID

retry: 15s # Duration until the server tries to re-authenticate after connection loss.

tls: # The Vault client TLS configuration for mTLS authentication and certificate verification

key: "" # Path to the TLS client private key for mTLS authentication to Vault

cert: "" # Path to the TLS client certificate for mTLS authentication to Vault

ca: "" # Path to one or multiple PEM root CA certificates

status: # Vault status configuration. The server will periodically reach out to Vault to check its status.

ping: 10s # Duration until the server checks Vault's status again.

Now, deploy the KES service along with the MinIO tenant:

$ kubectl apply -k operator/examples/kustomization/tenant-kes-encryption

This would deploy KES and MinIO pods as below

$ kubectl get pods -n tenant-kms-encrypted

NAME READY STATUS RESTARTS AGE

myminio-kes-0 1/1 Running 0 104m

myminio-kes-1 1/1 Running 0 104m

myminio-pool-0-0 2/2 Running 0 101m

myminio-pool-0-1 2/2 Running 0 102m

myminio-pool-0-2 2/2 Running 3 (102m ago) 103m

myminio-pool-0-3 2/2 Running 4 (102m ago) 104m

Your MinIO deployment with KES is now up and running.

Step 5: Verify the Deployment

To ensure that our deployment is running smoothly, let's check the status of the MinIO tenant pods, KES pods, and operator pods:

$ kubectl get pods -n minio-operator NAME READY STATUS RESTARTS AGE

console-6459d44b76-fgwbt 1/1 Running 0 5h29m

minio-operator-5668d46f98-9p2wm 1/1 Running 0 5h29m

minio-operator-5668d46f98-pm98s 1/1 Running 0 5h29m

$ kubectl get pods -n tenant-kms-encrypted

NAME READY STATUS RESTARTS AGE

myminio-kes-0 1/1 Running 0 116m

myminio-kes-1 1/1 Running 0 116m

myminio-pool-0-0 2/2 Running 0 113m

myminio-pool-0-1 2/2 Running 0 114m

myminio-pool-0-2 2/2 Running 3 (114m ago) 115m

myminio-pool-0-3 2/2 Running 4 (114m ago) 116m

$ k get pods -n default

NAME READY STATUS RESTARTS AGE

vault-0 1/1 Running 0 4h39m

Verify the KES pod logs to ensure that there are no errors and that the pods are running as expected as below:

$ kubectl logs myminio-kes-0 -n tenant-kms-encrypted Copyright MinIO, Inc. https://min.io

License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

Version 2023-04-18T19-36-09Z linux/amd64

KMS Hashicorp Vault: http://vault.default.svc.cluster.local:8200

Endpoints https://127.0.0.1:7373

https://10.244.3.55:7373

Admin _ [ disabled ]

Mem Lock off Failed to lock RAM pages. Consider granting CAP_IPC_LOCK

{"time":"2024-03-08T07:51:31.333681336Z","request":{"ip":"10.244.1.47","path":"/v1/key/create/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":400,"time":2871000}}

{"time":"2024-03-08T07:51:32.191375099Z","request":{"ip":"10.244.3.54","path":"/v1/identity/self/describe","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":18000}}

{"time":"2024-03-08T07:51:32.191784146Z","request":{"ip":"10.244.3.54","path":"/v1/key/create/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":400,"time":1739000}}

{"time":"2024-03-08T07:52:01.890935905Z","request":{"ip":"10.244.3.54","path":"/v1/key/generate/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":1888000}}

{"time":"2024-03-08T07:52:02.145905494Z","request":{"ip":"10.244.1.47","path":"/v1/key/generate/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":62000}}

{"time":"2024-03-08T07:52:02.162627602Z","request":{"ip":"10.244.1.47","path":"/v1/key/generate/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":62000}}

{"time":"2024-03-08T07:52:04.850766874Z","request":{"ip":"10.244.3.54","path":"/v1/key/decrypt/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":68000}}

{"time":"2024-03-08T07:52:04.857285115Z","request":{"ip":"10.244.3.54","path":"/v1/key/decrypt/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":53000}}

{"time":"2024-03-08T07:52:07.753077729Z","request":{"ip":"10.244.2.35","path":"/v1/identity/self/describe","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":7000}}

Step 6: Test the Deployment

To test our deployment, we will create a sample bucket and load/retrieve objects from a debug pod with mc (MinIO Client) installed.

First, start a new terminal and run the following command to exec into the debug pod:

kubectl exec -it pod/ubuntu-pod -n default -- bash

Inside the debug pod, run the following commands to create a bucket, create an encryption key, upload a file, and verify the encryption as demonstrated below:

mc alias ls myminio
mc ls myminio
mc mb myminio/encryptedbucket
mc admin kms key create myminio encrypted-bucket-key

echo "Hello" >> file1.txt
mc cp file1.txt myminio/encryptedbucket
mc ls myminio/encryptedbucket
mc cat myminio/encryptedbucket/file1.txt
mc admin kms key status myminio encrypted-bucket-key
mc stat myminio/encryptedbucket/file1.txt

If everything is set up correctly, you should see that the uploaded file is encrypted, and the encryption key status is valid.

The following demonstrates this process as expected:

$ kubectl exec -it pod/ubuntu-pod -n default -- bash

root@ubuntu-pod:/# mc alias ls myminio

myminio

URL : https://myminio-pool-0-0.myminio-hl.tenant-kms-encrypted.svc.cluster.local:9000

AccessKey : minio

SecretKey : minio123

API : s3v4

Path : auto

root@ubuntu-pod:/# mc ls myminio

root@ubuntu-pod:/# mc mb myminio/encryptedbucket

Bucket created successfully `myminio/encryptedbucket`.

root@ubuntu-pod:/# mc admin kms key create myminio encrypted-bucket-key

Created master key `encrypted-bucket-key` successfully

root@ubuntu-pod:/# echo "Hello" >> file1.txt

root@ubuntu-pod:/# mc cp file1.txt myminio/encryptedbucket

/file1.txt: 6 B / 6 B ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 344 B/s 0sroot@ubuntu-pod:/# mc ls myminio/encryptedbucket

[2024-03-08 10:25:01 UTC] 6B STANDARD file1.txt

root@ubuntu-pod:/# mc cat myminio/encryptedbucket/file1.txt

Hello

root@ubuntu-pod:/# mc admin kms key status myminio encrypted-bucket-key

Key: encrypted-bucket-key

- Encryption ✔

- Decryption ✔

root@ubuntu-pod:/# mc stat myminio/encryptedbucket/file1.txt

Name : file1.txt

Date : 2024-03-08 10:25:01 UTC

Size : 6 B

ETag : 27e775e1a5d22463e4cd39f12ce14ea0

Type : file

Metadata :

Content-Type: text/plain

Encrypted :

X-Amz-Server-Side-Encryption : aws:kms

X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:encrypted-bucket-key

If everything is set up correctly, you should see that the uploaded file is encrypted, and the encryption key status is valid, confirming that the object has been encrypted.

Final Thoughts

In this blog post, we explored how to deploy MinIO Operator with KES backed by Vault in a Kubernetes environment, with focus on Server-Side Encryption (SSE). By leveraging the power of Kubernetes and the kustomization plugin, we can automate the deployment process and easily scale our MinIO setup as needed.

Embracing a "shift left" approach, we underscore the importance of embedding security and data protection practices from the earliest stages of development and planning. The intent is clear: to elevate security from a mere consideration to a foundational element of your infrastructure. By spotlighting the straightforwardness and accessibility of SSE, our goal is to encourage its adoption as a standard across all MinIO deployments. The path to robust security is both simple and attainable, paving the way for a safer and more secure digital environment.

If you have questions, face any issues, or just want to talk about your experiences with MinIO, our community is ready to assist. Join us on Slack to meet other users, get advice from experts, and keep up with the latest news in object storage.

Happy encrypting!

Prerequisites

Step 1: Deploy a Kubernetes Cluster with kind

Step 2: Deploy MinIO Operator

Step 3: Set up HashiCorp Vault

Step 4: Deploy KES using kustomization plugin

Step 5: Verify the Deployment

Step 6: Test the Deployment

Final Thoughts

Step 4: Deploy KES using `kustomization` plugin