MinIO Batch Keyrotate

The Batch Framework feature in MinIO has been available since  RELEASE.2022-10-08T20-11-00Z and allows you to create, manage and execute jobs using a YAML formatted batch file. One of the cool things about the Batch Framework is that unlike the `mc` command line tool which runs on the client side, the Batch Framework allows you to use the server-side resources of the cluster where MinIO is deployed.

Each batch YAML file contains a single task that starts processing the job, and the MinIO Server monitors it until completion. If the batch job fails at any point during the run time, then it is retried up to N number of times as specified in the batch file.

At the moment, the MinIO Batch Framework supports the following two job types

We went into detail about batch replication in a previous blog post, so be sure to check that out. In this iteration we’ll talk about keyrotate and how to use it.

Why Rotate Keys?

The MinIO component that acts as the bridge between an external KMS and the application is MinIO’s Key Encryption Service (KES). KES is a stateless service that runs inside the Kubernetes cluster and distributes keys to applications. Due to the nature of the service being stateless it can be scaled and can also act like a cache so internal applications do not have to communicate with the external KMS every time. This reduces the overall load on the KMS and makes the application much more responsive since its interfacing locally with MinIO’s KES.

The SSE-S3 and SSE-KMS keys are used to encrypt objects in MinIO buckets when they are at rest. This provides key functionality for security, governance and compliance. MinIO’s current recommendation is to use your own external KMS (Key Management Service) such as Hashicorp’s Vault to manage your SSE keys. But, in the coming weeks, we are going to release stateful KES which will be just like a high-performance KMS built within MinIO that makes creation, deletion and management of the encryption fast and seamless.

The reason is that there are inevitably other applications and services in the infrastructure environment where keys need to be stored and managed in a secure manner. So putting our DevOps hats on, having each service’s keys managed individually is additional tech debt that could be a security risk if edge cases such as these are not considered. To keep things simple, we recommend using SSE in a stateless manner where it interfaces between the MinIO encryption layer and KMS but doesn’t actually end up storing any of the data (aka keys).

Internals of Batch Keyrotate

Batch keyrotate was released with MinIO version RELEASE.2023-04-07T05-28-58Z. In order to rotate the key we need to start the keyrotate batch job. But before we do that we need to point the keyrotate batch job to a batch YAMLfile with the required parameters as shown below.

This file can be generated using the following command

mc batch generate alias keyrotate

Which will generate a batch keyrotate.yaml file such as below

keyrotate:
apiVersion: v1
bucket: vbucket
prefix:
encryption:
type: sse-kms # valid values are sse-s3 and sse-kms
key: my-new-keys2 # valid only for sse-kms
# context: <new-kms-key-context> # valid only for sse-kms
# optional flags based filtering criteria
# for all objects
# flags:
#   filter:
#     newerThan: "84h" # match objects newer than this value (e.g. 7d10h31s)
#     olderThan: "80h" # match objects older than this value (e.g. 7d10h31s)
#     # createdAfter: "2023-03-02T15:04:05Z07:00" # match objects created after "date"
#     # createdBefore: "2023-03-02T15:04:05Z07:00" # match objects created before "date"
#     tags:
#       - key: "name"
#         value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
#     metadata:
#       - key: "content-type"
#         value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
#     kmskey: "key-id" # match objects with KMS key-id (applicable only for sse-kms)
#   notify:
#     endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
#     token: "Bearer xxxxx" # optional authentication token for the notification endpoint
#   retry:
#     attempts: 10 # number of retries for the job before giving up
#     delay: "500ms" # least amount of delay between each retry

Most of the above fields are self explanatory, but let's highlight the important ones.

Sometimes jobs can fail, so the batch file should also have a configured retry attempt and delay between those attempts.

How to Rotate the Key

This section explains how to rotate the key itself, but before we get started be sure the following prerequisites are met.

Prerequisites

• MinIO server with SSE-KMS enabled.
• mc admin alias set to the SSE-KMS enabled MinIO server.

Once the above prerequisites are met, go ahead and start the batch process configured in the previous YAML file using the command below

mc batch start alias/ ./keyrotate.yaml

You can list the currently submitted batch jobs

mc batch list alias/
ID                      TYPE            USER            STARTED
Qsk1r9YBkAuwZtMB5Sq4  keyrotate       minioadmin      4 minute ago

You can also check the status of the jobs

mc batch status alias/ Qsk1r9YBkAuwZtMB5Sq4

To check the configuration of batch jobs:

mc batch describe alias/ Qsk1r9YBkAuwZtMB5Sq4

Simplified and Automated Key Rotation

As you can see, encryption is an important part of the MinIO architecture. MinIO applies encryption to ensure objects are secure at rest and are compliant with regulations. We at MinIO have always thought about not only day-0 (design) and day-1 (implementation) phases of the architecture, but also day-2, which is the maintenance and ongoing operations aspect of it. Batch Key Rotation allows you to reduce the burden of day-2 operations of SSE-KMS keys – and decreases the tech debt in your software-defined infrastructure.

Benchmarks repeatedly show very little (<1%) throughput performance degradation with encryption turned on, therefore we recommend that all MinIO deployments use encryption at rest. While we’re making recommendations, all MinIO deployments should also secure network communications using TLS.

If you have any questions on how to use keyrotate be sure to reach out to us on Slack!