Introducing KES - Key Management at Scale

Andreas Auernhammer -

Today, the majority of data gets encrypted. Almost all applications use TLS / HTTPS to encrypt their network communication (in-flight) and encrypt the data separately before storing it (at-rest).

While it's interesting to look at how data gets encrypted, we want to focus on a different - sometimes considered an even harder - problem today:

How does the application get the encryption keys?

Key Management Systems

Typically, an application needs a new cryptographic key per data object. Therefore, an application usually requests a data encryption key (DEK) from a key management system (KMS). Traditionally, a KMS is a central component that maintains a set of master keys and stores them on a secure storage element - like FIPS-140 2 certified hardware security modules (HSMs).

When an application needs to encrypt some data, it can ask the KMS to derive a new DEK from a particular master key. Similarly, when an application needs to decrypt some data again, it can ask the KMS to decrypt the encrypted DEK with the same master key. This way the application never sees any master key - only plaintext/ciphertext DEK pairs - and the KMS does not need to store each and every encryption key.

An advanced KMS, like Hashicorp Vault, is well-suited for protecting your master keys. It also provides additional cryptographic APIs as well as other secret management features and can use certified hardware for storing secrets securely. Such a KMS can act as the root of trust in your infrastructure.

However, running and managing a full-featured KMS for large and distributed systems can be quite challenging, especially for high-performance applications. Traditional KMS solutions don't scale very well. For example, scaling a KMS hardware appliance means buying more expensive hardware. Further, full-featured KMS implementations are usually quite complex to configure and maintain - to the extend that KMS management requires a dedicated security team. This does match modern infrastructures running high-performance applications that are automatically adjusted to the current workload by Kubernetes.

When running a cloud-native infrastructure, you typically want the experience of a cloud KMS, like AWS-KMS, but on-prem.

KES

KES is a stateless and distributed key-management system for high-performance applications. It is designed to be run inside Kubernetes and distribute cryptographic keys to applications. To be clear, KES cannot and does not try to replace a full-featured KMS. Instead, it acts as a bridge between a central KMS and cloud-native applications.

So, there is still one central KMS protecting master keys and acting as the root of trust in your infrastructure. However, you don't need to deploy and manage one KMS per set of applications. Instead, an application can request a new DEK from a KES server or ask the KES server to decrypt an encrypted DEK.

Since the KES server is completely stateless it can be scaled automatically - for example via the K8S horizontal autoscaler. At the same time the load on the central KMS does not increase much because KES can serve the vast majority of application requests without talking to the KMS.

Keep it simple

As mentioned before, KES is not and doesn't try to be a full-featured KMS implementation. Instead, it focuses on a few but very common key management operations and exposes them via an easy to consume REST API - cURL is all you need.

However, a simple API is just one aspect. It is also very easy to configure and manage a KES server - especially compared to traditional KMS implementations. The KES server is a static binary that consumes a yaml config file. Moving a setup from a developer laptop to a Kubernetes cluster is a friction-less experience.

But a simple design doesn't just make the development and operation aspects easier. It also helps keeping the system secure.

Keep it secure

KES is designed to be secure by default. For instance, KES can only be run with TLS enabled. While this may sound, initially, like a limitation it actually has some neat implications.

First of all, all setups use secure communication and a potential development or testing environment doesn't differ from production with regard to TLS. By requiring TLS, KES can also completely rely on TLS and X.509 certificates for authentication as well as authorization.  No additional username/password, JWT or other authentication mechanism is needed. TLS is the universal authentication mechanism on the Internet such that almost everyone supports it - regardless of the underlying operating system or programming language.

We are aware, however, that it can be quite challenging to wrap your head around TLS - especially when it comes to certificates. Therefore, we tried to make the tooling abstract the low-level details. As with almost everything MinIO, you can, if you choose, go deep into the weeds, but you should not need to.

Instead of explaining all the details here, let's get started with some examples.

Getting started

If you want to, you can setup a local KES server in a minute by following our getting started guide. However, we also run a public KES server instance at https://play.min.io:7373 for you to try it out and experiment.

As an initial step you need to fetch the private key and certificate of the root identity:

curl -sSL --tlsv1.2 \
   -O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
   -O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'

Now, you can directly start using the KES server API. For example you can create a new master key named my-key - if it doesn't exist already - via:

curl -sSL --http2 --tlsv1.3 \
   --key root.key \
   --cert root.cert \
   -X POST 'https://play.min.io:7373/v1/key/create/my-key'

As last example, we request a new DEK plaintext/ciphertext pair from the my-key master key via:

curl -sSL --http2 --tlsv1.3 \
   --key root.key \
   --cert root.cert \
   --data '{}' \
   -X POST 'https://play.min.io:7373/v1/key/generate/my-key'

For further experiments you may want to use the KES CLI and point it to our play instance:

export KES_SERVER=https://play.min.io:7373
export KES_CLIENT_KEY=root.key
export KES_CLIENT_CERT=root.cert

You may want to monitor what's going on there:

kes log trace

What's next

KES is an open source project licensed under AGPLv3. You can find it on Github. Currently, KES can use Hashicorp Vault and AWS SecretsManager + AWS-KMS as the central KMS. If you want to use KES with another KMS implementation or have a feature request, just open an issue.

Further, our object storage server already relies on KES to implement S3 server-side encryption with a KMS. To learn more about how KES works, checkout our Concepts page or the KES server API documentation. If you have further questions, join our legendary community Slack channel - we will help you on your way.