Secrets Made Easy with MinIO and HashiCorp Vault

Enterprises require security to protect systems, with a primary focus on data at rest and data in transit. There are many facets to a viable security program, and this blog post focuses on cryptographic key management. Encryption and associated keys are what keeps data safe. For example, websites require a certificate handshake between client and server (in this case, browser and the web/app server). On the back end, managing multiple environments and many microservices in a polyglot architecture requires that credentials be managed effectively. Data stored in object storage solutions may potentially need to meet compliance requirements and be protected, and each object needs to be encrypted and stored, as well as decrypted when required. For these reasons, infrastructure and security teams require a solid vaulting solution.

Products/Applications need to encrypt data: Products/Applications do not need to complicate their implementation with cryptography when they can just rely on Hashicorp’s Vault to provide this capability. Vault focuses on signing and verifying data, not storing it. The combination of applications such as MinIO and Vault will ensure data at rest security requirements are met.

Protect databases and web services credentials:  Microservices architecture with polyglot databases and integrations with multiple web services or microservices is commonplace. In addition, there are always multiple environments. With infrastructure-as-code and automation, managing credentials and access keys becomes essential, making it necessary to have a high-performing key management and storage solution such as Vault and KES.

Certificates for data in transit: Whether it is a website accessed from a browser, communication between applications and databases, from applications to web services or between microservices, the communication channels between each must be encrypted. Vault can generate PKI certificates that can be used to secure data in transit in these channels.

In order to provide functionality for regulatory compliance around secure locking and erasure, MinIO encrypts objects at the storage layer by using Server-Side Encryption (SSE) to protect objects as part of write operations. MinIO does this with extreme efficiency – benchmarks show that MinIO is capable of encrypting/decrypting at close to wire speed.

How does MinIO achieve this level of efficiency and performance?

The secret sauce MinIO uses is Single Instruction Multiple Data (SIMD). Generally, you can only send one CPU instruction at a time and wait for a response before sending the next instruction. This is highly inefficient, especially when performing thousands – if not millions – of encryption and decryption instructions per second. MinIO takes advantage of SIMD so it can send multiple instructions in a single request to be processed by the CPU. We’ve written this in assembly within GoLang so we are as close to the hardware layer as possible to take full advantage of the typically underutilized  CPU power to perform cryptographic operations at scale.

HashiCorp’s Vault shares a lot of the same thought processes and design values as MinIO – resulting in complementary software that is both powerful and a pleasure to use. For example, Vault can run in almost any environment. It is cloud agnostic, which means it can run on AWS, GCP, Azure and even bare metal. It's very easy to get up and running as Vault is just a single binary. If you are developing on your laptop, you can run it in dev mode, and when you are ready to move to production you can scale it out across multiple nodes to form a cluster.

Vault is also very versatile with several use cases. One such use case is managing the keys for LUKS (Linux Unified Key Setup) which generally stores the keys on the same volume where encryption and decryption operations are performed. Obviously, this is a huge security concern because the key is stored alongside the data. Instead we can query the Vault service on bootup to fetch the keys required for LUKS to perform cryptographic operations. This way even if the data becomes compromised, the keys will not be on the volume and thus there is no way to decrypt the data short of exploiting Vault.

MinIO’s Key Encryption Service (KES) brings this all together. SSE uses KES and Vault to perform cryptographic operations. The KES service itself is stateless and acts as a middle layer as it stores its data in Vault. With MinIO you can set up various levels of granular and customizable encryption. You can always choose to encrypt on a per object basis, however, we strongly recommend setting up SSE-KMS encryption automatically on the buckets so all objects are encrypted by default. Encryption is accomplished using a specific External Key (EK) stored in Vault, which can be overridden on a per object basis with a unique key.

In this blog post we’ll show you how you can quickly get up and running with MinIO, KES and Vault to fully understand the capabilities of server-side encryption.

Set up Infrastructure

To get started, launch a Linux VM on your laptop or in the cloud where we can install these components. SSH into this VM and ensure you are performing all operations as either root user or using sudo. In this blog post we are using a vanilla Ubuntu VM.

MinIO

We’ll set up this configuration to launch a single-node single-drive setup to allow us to visualize the concepts a bit better. You can easily expand this tutorial to a distributed setup.

Download and move MinIO binary to an executable location

# wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20221126224332.0.0_amd64.deb -O minio.deb

# dpkg -i minio.deb

Install the MinIO client as well

# wget https://dl.min.io/client/mc/release/linux-amd64/mc
# chmod +x mc
# mv mc /usr/local/bin/mc

Once you install the package onto the VM, do not start the MinIO service just yet. We need to set up a couple of additional prerequisites before we start it.

KES

KES runs as its own service and is the intermediary between MinIO and the Vault service.

Setup the following directories on the VM to add the certificates we generate.

# mkdir -p /opt/kes/certs
# mkdir -p /opt/kes/config
# mkdir -p /opt/minio/certs
# mkdir -p /opt/minio/config
# mkdir -p ~/minio

Fetch the KES binary from the MinIO KES repo

# wget https://github.com/minio/kes/releases/download/v0.22.1/kes-linux-amd64

Chmod the KES binary and move it to a PATH executable location

# chmod +x kes-linux-amd64
# mv kes-linux-amd64 /usr/local/bin/kes

# kes --version

Once again, do not start the KES server just yet, we still have a few things we need to configure before we can start it.

Vault

KES requires Vault to be running and unsealed before it can communicate with it.

Let’s install Vault using the steps below

Open a new tmux session to run the Vault operations

# tmux new -s vault

Install the GPG package for adding apt keys

# apt update && apt install gpg

Fetch the Hashicorp apt repo keys

# wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg >/dev/null

Verify the fingerprint

# gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

Add the Hashicorp apt repo so we can install the Vault package

# echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

Last, but not least, install Vault itself

# apt update && apt install vault

Start Vault server, which will also unseal it for us

# vault server -dev

Once Vault is up, note the Vault Endpoint and the Vault Root Token. You will need these values later to perform operations within Vault.

$ export VAULT_ADDR='http://127.0.0.1:8200'

[TRUNCATED]

Root Token: hvs.rCFo4tdgIdiq5NTRo6VzbBGz

End the tmux session using the following keystrokes

CTRL+B then press D

Configure Infrastructure

Once we have the infrastructure set up, we’ll need to configure individual components

Vault

Outside the Vault TMUX session, set the following environment variables

VAULT_ADDR

VAULT_TOKEN

The values for these can be found in the earlier output when the Vault service was started.

# export VAULT_ADDR='http://127.0.0.1:8200'

# export VAULT_TOKEN="hvs.rCFo4tdgIdiq5NTRjrVzbBGz"

Create a Vault secret engine path called kv/

# vault secrets enable -path=kv kv

Success! Enabled the kv secrets engine at: kv/

Enable the Vault app role to support KES. This is used for KES to authenticate with the Vault app by assigning the required permissions and retrieving App ID and Secret for KES.

# vault auth enable approle

Success! Enabled approle auth method at: approle/

Create a file called kes-policy.hcl with the following contents in order to provide the necessary access to the kv/ engine we created earlier.

path "kv/data/*" {
capabilities = [ "create", "read"]
}

path "kv/metadata/*" {
capabilities = [ "list", "delete"]
}

Apply the file above to create a policy in Vault

# vault policy write kes-policy kes-policy.hcl

Success! Uploaded policy: kes-policy

Create an app role called kes-role and assign it the policy we created in the previous step

# vault write    auth/approle/role/kes-role token_num_uses=0 secret_id_num_uses=0 period=5m
Success! Data written to: auth/approle/role/kes-role

# vault write    auth/approle/role/kes-role policies=kes-policy
Success! Data written to: auth/approle/role/kes-role

KES

Create a TLS certificate for KES to secure communications between KES and the Vault deployment

# kes identity new kes_server \
--key  /opt/kes/certs/kes-server.key  \
--cert /opt/kes/certs/kes-server.cert  \
--ip   "127.0.0.1"  \
--dns  localhost

Next, we’ll set a TLS certificate for MinIO to perform mTLS authentication to KES

# kes identity new minio_server \
--key  /opt/minio/certs/minio-kes.key  \
--cert /opt/minio/certs/minio-kes.cert  \
--ip   "127.0.0.1"  \
--dns  localhost

Create a KES configuration file with the following contents /opt/kes/config/kes-config.yaml. We’ll explain how to get the required information to fill out the configuration.

address: 0.0.0.0:7373

admin:
  identity: disabled

tls:
  key:  /opt/kes/certs/kes-server.key
  cert: /opt/kes/certs/kes-server.cert

policy:
  minio:
    allow:
    - /v1/key/create/*   
    - /v1/key/generate/* # e.g. '/minio-'
    - /v1/key/decrypt/*
    identities:
    - MINIO_IDENTITY_HASH
keystore:
  vault:
    endpoint: http://localhost:8200
    engine: "kv/" # Replace with the path to the K/V Engine
    version: "v2" # Specify v1 or v2 depending on the version of the K/V Engine
    approle:
      id: "VAULTAPPID"     # Hashicorp Vault AppRole ID
      secret: "VAULTAPPSECRET" # Hashicorp Vault AppRole Secret ID
      retry: 15s
    status:
      ping: 10s

Here’s how you obtain the necessary information.

Fetch MINIO_IDENTITY_HASH using the following command

# kes identity of /opt/minio/certs/minio-kes.cert

Get VAULTAPPID and VAULTAPPSECRET using the following commands

The keys from our output look something like this

# vault read     auth/approle/role/kes-role/role-id

Key        Value
---        -----
role_id    6924d20f-31b3-xxxx-6182-5fce1fd52d37

# vault write -f auth/approle/role/kes-role/secret-id

Key                   Value
---                   -----
secret_id             0d6d8576-6a74-8f7f-xxx-785a5ec6fb3d
secret_id_accessor    16442cd9-8ba6-e730-042a-6ff756f2947f
secret_id_num_uses    0
secret_id_ttl         0s

MinIO

Last, but not least, configure MinIO

In /opt/minio/config/minio add the following config

MINIO_KMS_KES_ENDPOINT=https://localhost:7373
MINIO_KMS_KES_CERT_FILE=/opt/minio/certs/minio-kes.cert
MINIO_KMS_KES_KEY_FILE=/opt/minio/certs/minio-kes.key

MINIO_KMS_KES_CAPATH=/opt/kes/certs/kes-server.cert

MINIO_KMS_KES_KEY_NAME=minio-backend-default-key

Open a new tmux session to start KES service

# tmux new -s kes

Start the KES service using the kes-config.yaml file we created earlier

# setcap cap_ipc_lock=+ep $(readlink -f $(which kes))

# kes server --auth=off --config=/opt/kes/config/kes-config.yaml

Detach out of the tmux session using the following keystrokes

CTRL+B then press D

Now that we have all the additional services needed for KES, let's start the MinIO service.

Open a new tmux session to start the MinIO service using the config file we created earlier

# tmux new -s minio

# export MINIO_CONFIG_ENV_FILE=/opt/minio/config/minio
# minio server ~/minio --console-address :9001

Once MinIO is started, detach from the tmux session using the following keystrokes

CTRL+B then press D

Testing MinIO, KES and Vault

At this point, you should have the necessary components to get SSE working. Let's be good netizens and test it.

Generate a new encrypted bucket key

# export KES_SERVER=https://127.0.0.1:7373
# export KES_CLIENT_KEY=/opt/minio/certs/minio-kes.key
# export KES_CLIENT_CERT=/opt/minio/certs/minio-kes.cert

# kes key create -k encrypted-bucket-key

Create a bucket in MinIO and enable auto encryption for every object added to the bucket by default.

# mc alias set local http://127.0.0.1:9000 minioadmin minioadmin

# mc mb local/encryptedbucket

Bucket created successfully `local/encryptedbucket`.

# mc encrypt set SSE-KMS encrypted-bucket-key local/encryptedbucket

Auto encryption configuration has been set successfully for local/encryptedbucket

Add an object to bucket using cp

# mc cp file.txt local/encryptedbucket/file.txt

/root/file.txt:  5 B / 5 B ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161 B/s 0s

Finally the pièce de résistance. Verify that the object we just added has been encrypted. We can do this using the stat command.

# mc stat local/encryptedbucket/file.txt

Name      : file.txt
Date      : 2022-11-28 19:30:05 UTC
Size      : 5 B
ETag      : add93d98cf5931d7642118efded8aa20
Type      : file
Metadata  :
Content-Type: text/plain
Encrypted :
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:encrypted-bucket-key
X-Amz-Server-Side-Encryption               : aws:kms

You can also verify by listing all Vault keys in the kv/ engine.

# vault kv list kv/data

Keys
----
encrypted-bucket-key
minio-backend-default-key

If you see outputs similar to above then you have successfully configured SSE. That wasn’t so hard was it? Now you know why and how to encrypt data stored in MinIO.

Final Thoughts

This blog post focused on Server-Side Encryption (SSE), Key Encryption Service (KES) and Key Management System (KMS, using Vault). We showed you how simple it is to get started with SSE, and that the cryptographic benefits far outweigh the level of effort (LOE) needed to set up additional components. You may now “shift left”  and implement security best practices early on in the development and planning process; security and data protection should never be an afterthought. By focusing on SSE and how simple it is to use and understand, we hope that you are able to implement this as a standard for all your MinIO deployments.

What you’ve just run through is an introduction to MinIO-KES-Vault. To evolve your learning environment into a production environment,  you must ensure you are following TLS best practices by proper valid certificates from a trusted CA (such as LetsEncrypt). Use different certificates to secure communications between MinIO -> KES and KES -> Vault. Similar to MinIO in multi-server multi-drive setup, KES should also be set up on multiple nodes using the instructions here. The same applies to Vault where you should be using industry best practices to set up a production distributed setup that is not on the same node as the rest of the MinIO infrastructure since there is a potential to use Vault with other services as well.

We hope you found this tutorial blog post useful in understanding this important topic. As usual, if you have any questions please feel free to reach out to us on Slack!