MinIO Operator with KES backed by Vault
In order to provide functionality for regulatory compliance around secure locking and erasure, MinIO encrypts objects at the storage layer by using Server-Side Encryption (SSE) to protect objects as part of write operations. MinIO does this with extreme efficiency – benchmarks show that MinIO is capable of encrypting/decrypting at close to wire speed.
The secret sauce MinIO uses is Single Instruction Multiple Data (SIMD). Generally, you can only send one CPU instruction at a time and wait for a response before sending the next instruction. This is highly inefficient, especially when performing thousands – if not millions – of encryption and decryption instructions per second. MinIO takes advantage of SIMD so it can send multiple instructions in a single request to be processed by the CPU. We’ve written this in assembly within GoLang so we are as close to the hardware layer as possible to take full advantage of the typically underutilized CPU power to perform cryptographic operations at scale.
In a previous post we showed you how to get started with MinIO and Vault in a bare-metal environment using KES. In this post we’ll show you how to configure MinIO Operator with KES (Keys Encryption System) and Vault in a cloud native way in Kubernetes. This will enable you to automate the process as you scale and use Kubernetes resources to configure them.
Prerequisites
Before we begin, ensure that you have the following prerequisites in place:
- A Kubernetes cluster (for this tutorial, we will use kind to create a local cluster)
- kubectl command-line tool installed on your local machine
- git for cloning the necessary repositories
Step 1: Deploy a Kubernetes Cluster with kind
To get started, we will create a Kubernetes cluster using kind. This will provide us with a local environment to deploy and test our MinIO setup.
First, create a kind configuration file named kind-config.yaml with the following command:
$ cat > kind-config.yaml <<EOF kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker - role: worker - role: worker EOF $ kind create cluster --config kind-config.yml |
Step 2: Deploy MinIO Operator
With our Kubernetes cluster up and running, we can now deploy the MinIO Operator using the kustomization plugin. The MinIO Operator simplifies the deployment and management of MinIO instances in a Kubernetes environment.
Execute the following command to deploy the MinIO Operator in kind cluster:
$ kubectl apply -k github.com/minio/operator |
Wait for all the pods to come online in the minio-operator namespace before proceeding to the next step.
Step 3: Set up HashiCorp Vault
HashiCorp Vault is a powerful secrets management tool that we will use to securely store and manage our encryption keys. To set up Vault in our Kubernetes cluster, we will use the kubernetes-vault repository.
Clone the repository by running the following command:
$ git clone https://github.com/scriptcamp/kubernetes-vault.git $ cd kubernetes-vault/vault-manifests |
Next, set up the necessary RBAC (Role-Based Access Control) rules, create Vault config maps, deploy Vault services, and set up a stateful set:
$ kubectl apply -f rbac.yml |
Create Vault config maps
$ kubectl apply -f configmap.yaml |
Deploy Vault services
$ kubectl apply -f services.yaml |
As Vault is stateful service, we need to set up a stateful set for the same. Execute the below command for the same
$ kubectl apply -f statefulset.yaml |
Once the Vault setup is complete, unseal and initialize Vault.
$ kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > keys.json $ VAULT_UNSEAL_KEY=$(cat keys.json | jq -r ".unseal_keys_b64[]") $ echo $VAULT_UNSEAL_KEY $ VAULT_ROOT_KEY=$(cat keys.json | jq -r ".root_token") $ echo $VAULT_ROOT_KEY $ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY |
Now, enter the vault-0 POS to perform additional configuration:
$ kubectl exec -it vault-0 -- /bin/sh |
Inside the pod, enable the K/V backend, create a policy, enable AppRole authentication, create a KES role, and generate an app-role ID and secret:
$ vault secrets enable -version=1 kv |
Create policy
$ cat > kes-policy.hcl <<EOF path "kv/*" { capabilities = [ "create", "read", "delete" ] } $ vault policy write kes-policy kes-policy.hcl |
Enable AppRole authentication
$ vault auth enable approle |
Create KES role and attach policy to it
$ vault write auth/approle/role/kes-server token_num_uses=0 secret_id_num_uses=0 period=5m $ vault write auth/approle/role/kes-server policies=kes-policy |
Generate app-role ID and secret, and take a note of values
$ vault read auth/approle/role/kes-server/role-id Key Value --- ----- role_id b484633b-8965-08dd-0e24-d6973e6be2d2 $ vault write -f auth/approle/role/kes-server/secret-id Key Value --- ----- secret_id 7fd44d44-a3f1-a013-1f40-e1952496c416 secret_id_accessor b5ee0c97-5ffc-b9a7-fe0b-763757fe1033 secret_id_ttl 0s |
Take note of the generated app-role ID and secret, as we will need them in the next step.
Step 4: Deploy KES using kustomization plugin
To deploy KES (Key Encryption Service) using the kustomization plugin, we first need to clone the MinIO repository:
$ git clone https://github.com/minio/operator.git $ cd operator |
Update the examples/kustomization/tenant-kes-encryption/kes-configuration-secret.yaml file with the appropriate KES values, including the Vault endpoint, namespace, prefix, and the generated app-role ID and secret.
keystore: ## KES configured with fs (File System mode) doesnt work in Kubernetes environments and it's not recommended ## use a real KMS # fs: # path: "./keys" # Path to directory. Keys will be stored as files. Not Recommended for Production. vault: endpoint: "http://vault.default.svc.cluster.local:8200" # The Vault endpoint namespace: "default" # An optional Vault namespace. See: https://www.vaultproject.io/docs/enterprise/namespaces/index.html prefix: "my-minio" # An optional K/V prefix. The server will store keys under this prefix. approle: # AppRole credentials. See: https://www.vaultproject.io/docs/auth/approle.html id: "b484633b-8965-08dd-0e24-d6973e6be2d2" # Your AppRole Role ID secret: "7fd44d44-a3f1-a013-1f40-e1952496c416" # Your AppRole Secret ID retry: 15s # Duration until the server tries to re-authenticate after connection loss. tls: # The Vault client TLS configuration for mTLS authentication and certificate verification key: "" # Path to the TLS client private key for mTLS authentication to Vault cert: "" # Path to the TLS client certificate for mTLS authentication to Vault ca: "" # Path to one or multiple PEM root CA certificates status: # Vault status configuration. The server will periodically reach out to Vault to check its status. ping: 10s # Duration until the server checks Vault's status again. |
Now, deploy the KES service along with the MinIO tenant:
$ kubectl apply -k operator/examples/kustomization/tenant-kes-encryption |
This would deploy KES and MinIO pods as below
$ kubectl get pods -n tenant-kms-encrypted NAME READY STATUS RESTARTS AGE myminio-kes-0 1/1 Running 0 104m myminio-kes-1 1/1 Running 0 104m myminio-pool-0-0 2/2 Running 0 101m myminio-pool-0-1 2/2 Running 0 102m myminio-pool-0-2 2/2 Running 3 (102m ago) 103m myminio-pool-0-3 2/2 Running 4 (102m ago) 104m |
Your MinIO deployment with KES is now up and running.
Step 5: Verify the Deployment
To ensure that our deployment is running smoothly, let's check the status of the MinIO tenant pods, KES pods, and operator pods:
$ kubectl get pods -n minio-operator NAME READY STATUS RESTARTS AGE console-6459d44b76-fgwbt 1/1 Running 0 5h29m minio-operator-5668d46f98-9p2wm 1/1 Running 0 5h29m minio-operator-5668d46f98-pm98s 1/1 Running 0 5h29m $ kubectl get pods -n tenant-kms-encrypted NAME READY STATUS RESTARTS AGE myminio-kes-0 1/1 Running 0 116m myminio-kes-1 1/1 Running 0 116m myminio-pool-0-0 2/2 Running 0 113m myminio-pool-0-1 2/2 Running 0 114m myminio-pool-0-2 2/2 Running 3 (114m ago) 115m myminio-pool-0-3 2/2 Running 4 (114m ago) 116m $ k get pods -n default NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 4h39m |
Verify the KES pod logs to ensure that there are no errors and that the pods are running as expected as below:
$ kubectl logs myminio-kes-0 -n tenant-kms-encrypted Copyright MinIO, Inc. https://min.io License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html Version 2023-04-18T19-36-09Z linux/amd64 KMS Hashicorp Vault: http://vault.default.svc.cluster.local:8200 Endpoints https://127.0.0.1:7373 https://10.244.3.55:7373 Admin _ [ disabled ] Mem Lock off Failed to lock RAM pages. Consider granting CAP_IPC_LOCK {"time":"2024-03-08T07:51:31.333681336Z","request":{"ip":"10.244.1.47","path":"/v1/key/create/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":400,"time":2871000}} {"time":"2024-03-08T07:51:32.191375099Z","request":{"ip":"10.244.3.54","path":"/v1/identity/self/describe","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":18000}} {"time":"2024-03-08T07:51:32.191784146Z","request":{"ip":"10.244.3.54","path":"/v1/key/create/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":400,"time":1739000}} {"time":"2024-03-08T07:52:01.890935905Z","request":{"ip":"10.244.3.54","path":"/v1/key/generate/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":1888000}} {"time":"2024-03-08T07:52:02.145905494Z","request":{"ip":"10.244.1.47","path":"/v1/key/generate/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":62000}} {"time":"2024-03-08T07:52:02.162627602Z","request":{"ip":"10.244.1.47","path":"/v1/key/generate/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":62000}} {"time":"2024-03-08T07:52:04.850766874Z","request":{"ip":"10.244.3.54","path":"/v1/key/decrypt/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":68000}} {"time":"2024-03-08T07:52:04.857285115Z","request":{"ip":"10.244.3.54","path":"/v1/key/decrypt/my-minio-key","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":53000}} {"time":"2024-03-08T07:52:07.753077729Z","request":{"ip":"10.244.2.35","path":"/v1/identity/self/describe","identity":"f9daf353b4bb9bba772a369b621a258c9464322c85d1ac9d2aafd29afdb96ea6"},"response":{"code":200,"time":7000}} |
Step 6: Test the Deployment
To test our deployment, we will create a sample bucket and load/retrieve objects from a debug pod with mc (MinIO Client) installed.
First, start a new terminal and run the following command to exec into the debug pod:
kubectl exec -it pod/ubuntu-pod -n default -- bash |
Inside the debug pod, run the following commands to create a bucket, create an encryption key, upload a file, and verify the encryption as demonstrated below:
mc alias ls myminio |
If everything is set up correctly, you should see that the uploaded file is encrypted, and the encryption key status is valid.
The following demonstrates this process as expected:
$ kubectl exec -it pod/ubuntu-pod -n default -- bash root@ubuntu-pod:/# mc alias ls myminio myminio URL : https://myminio-pool-0-0.myminio-hl.tenant-kms-encrypted.svc.cluster.local:9000 AccessKey : minio SecretKey : minio123 API : s3v4 Path : auto root@ubuntu-pod:/# mc ls myminio root@ubuntu-pod:/# mc mb myminio/encryptedbucket Bucket created successfully `myminio/encryptedbucket`. root@ubuntu-pod:/# mc admin kms key create myminio encrypted-bucket-key Created master key `encrypted-bucket-key` successfully root@ubuntu-pod:/# echo "Hello" >> file1.txt root@ubuntu-pod:/# mc cp file1.txt myminio/encryptedbucket /file1.txt: 6 B / 6 B ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 344 B/s 0sroot@ubuntu-pod:/# mc ls myminio/encryptedbucket [2024-03-08 10:25:01 UTC] 6B STANDARD file1.txt root@ubuntu-pod:/# mc cat myminio/encryptedbucket/file1.txt Hello root@ubuntu-pod:/# mc admin kms key status myminio encrypted-bucket-key Key: encrypted-bucket-key - Encryption ✔ - Decryption ✔ root@ubuntu-pod:/# mc stat myminio/encryptedbucket/file1.txt Name : file1.txt Date : 2024-03-08 10:25:01 UTC Size : 6 B ETag : 27e775e1a5d22463e4cd39f12ce14ea0 Type : file Metadata : Content-Type: text/plain Encrypted : X-Amz-Server-Side-Encryption : aws:kms X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:encrypted-bucket-key |
If everything is set up correctly, you should see that the uploaded file is encrypted, and the encryption key status is valid, confirming that the object has been encrypted.
Final Thoughts
In this blog post, we explored how to deploy MinIO Operator with KES backed by Vault in a Kubernetes environment, with focus on Server-Side Encryption (SSE). By leveraging the power of Kubernetes and the kustomization plugin, we can automate the deployment process and easily scale our MinIO setup as needed.
Embracing a "shift left" approach, we underscore the importance of embedding security and data protection practices from the earliest stages of development and planning. The intent is clear: to elevate security from a mere consideration to a foundational element of your infrastructure. By spotlighting the straightforwardness and accessibility of SSE, our goal is to encourage its adoption as a standard across all MinIO deployments. The path to robust security is both simple and attainable, paving the way for a safer and more secure digital environment.
If you have questions, face any issues, or just want to talk about your experiences with MinIO, our community is ready to assist. Join us on Slack to meet other users, get advice from experts, and keep up with the latest news in object storage.
Happy encrypting!