Erasure Code Calculator
Reference Hardware
Renewing KES certificate
MiniO KES (Key Encryption Service) is a service developed by MinIO to bridge the gap between applications that run in Kubernetes and a centralized Key Management Service (KMS). The central KMS server contains all the state information, while the KES talks to the KMS when it is required to do any operation related to fetching new keys or updating existing ones. Once it fetches a key, as long as it doesn’t need to be updated or deleted, it will be cached in KES so the subsequent calls will be much faster.
So why use KES rather than directly using the KMS? Depending on the KMS used and the load it needs to handle, sometimes KMS systems do not have the capability or the support to handle large deployments where it has to manage hundreds if not thousands of keys back and forth while the Kubernetes cluster puts an enormous load on them. In these situations, it's paramount you use KES because it can scale horizontally very easily, unlike traditional KMS systems.
All KES operations between the Application <-> KES and between KES <-> KMS use mTLS authentication for authentication and authorization functions. This is done using a pair of public/private keys and X.509 certificate. The thing with certs is they have a very common problem, they tend to expire and when they do, services all around fail spectacularly with little rhyme or reason. What do we mean by that?
What we mean is once the cert expires, you will start to see errors such as these in the KES log
Also, when MinIO tries to do a periodic IAM refresh, those would also fail with the following messages in the MinIO log
If you are lucky, you will see an obvious message such as certificate has expired. Other times it's not so obvious, you could also see edge case issues when trying to create or delete keys among a host of other issues. The quickest solution is to renew and update KES with new certs as soon as possible. In this post we’ll show you exactly how to do that.
How to Renew
Let’s first start by creating a new private key
Create a file called cert.cnf which will be used by openssl to create the Certificate Signing Request (CSR)
Be sure to modify the Common Name CN and Subject Alternative Names SAN (under [alt_names]) to match the FQDN of your KES nodes. Be sure to use proper FQDNs and not IP addresses.
Create the CSR using the command below
Convert the CSR into an encoded string so it can be added to Kubernetes as a CertificateSigningRequest resource.
Create a file kes-csr.yaml with the content below and paste the above encoded CSR in the request field. The cert has been truncated so you can see the entire yaml.
Be sure to update the expirationSeconds to something high so that it doesn’t expire very soon.
Once the encoded CSR has been added and other settings have been set, apply the yaml.
Be sure to approve the kes-csr CSR created above
Get a public cert from the csr resource
Convert both the private.key (from the beginning of the process) and public.crt (from the previous step) to an encoded string.
Using the encoded strings from above, we’ll update the existing Secret kes-tenant-kes-tls, in order to do that, follow the steps below.
Copy the existing secret where the existing expired cert is located.
Once you have backed up the existing secret, delete it
Open kes-tls-secret.yaml with the expired certs and replace the following two fields with their respective base64 encoded strings.
Once the new certs have been added apply the Secret, which will recreate kes-tenant-kes-tls
Once a valid cert is added, be sure to restart the KES service and you should see the output like so:
The MinIO log should also be clean and should not show any TLS errors anymore.
Final Thoughts
KES is an integral part when it comes to managing keys used to encrypt objects. It is important that objects get encrypted and decrypted in the quickest manner possible because each nanosecond it takes to perform these operations the end user will get the objects that much slower. Yes, ultimately, inefficient and slow KMS systems can degrade overall cluster performance. So it's paramount to ensure the service performing these actions is fast, lean, performant and scalable. MinIO’s KES enables any KMS to be a high performance and scalable service without any modification to the existing KMS. By following the steps above, you can get KES back to having valid unexpired certs in no time!
If you have any questions on KES be sure to reach out to us on Slack!