Streamline Certificate Management with MinIO Operator

Modern, cloud-native architectures are build around modern object storage. MinIO and AWS S3 are excellent examples of performant, scalable, resilient data stores that excel at serving data to applications and users.

Security is top of mind for everyone and sensitive data must be protected from attackers that may want to read it and/or alter it, especially as it traverses external networks. Transport Layer Security (TLS) v1.2+ is the most common method of encrypting data in-transit in the cloud-native world.

However, numerous challenges must be overcome to successfully configure and manage certificates under Kubernetes. A recent survey found that more than 60 percent of CIOs reported that they experienced certificate-related outages within the last 12 months. Generating certificates requires knowledge of the system's structure and a keen attention to detail. Configuring TLS is detail-oriented and therefore error-prone. Once configured, certificates must then be applied correctly and renewed on time in order to prevent system disruption.  

The MinIO Operator simplifies and streamlines the generation and assignment of TLS certificates for MinIO components running on Kubernetes. MinIO tenants can use multiple certificates, where each certificate corresponds to a domain name that maps to a specific bucket. Use the MinIO Operator to manage certificates whether they are automatically generated using TLS Certificate Signing Requests (CSR) and the Kubernetes certificates.k8s.io TLS certificate management API or they are generated by external third party services or software.  

MinIO extends the S3 security spec to include encryption of data in-transit over the network and at-rest saved to drives. MinIO relies on TLS to protect sensitive data, such as credentials and objects, as it is transmitted over the network.  

In this post, you will learn how to manage certificates using MinIO Operator and Kubernetes.

Getting Started Managing Certificates with MinIO Operator

Before you start this tutorial, you’ll need to have Kubernetes v1.21 or higher installed. You will also need the kubectl CLI tool and krew plugin manager. In addition, the Kubernetes TLS certificate API must be configured as described in Deploy the MinIO Operator — MinIO Object Storage for Kubernetes.  

These instructions are for working with MinIO v5 and higher. If you don’t already have MinIO Operator installed, you’ll have to install it

This will deploy the MinIO Operator into the minio-operator namespace. Deployment includes the MinIO Operator and its dependencies, including the automatically generated certificates needed for MinIO Operator and MinIO Console stored as Kubernetes secrets.

You can verify this by examining logs for the pods in the minio-operator namespace. Below is the log from the MinIO Operator pod, note that when the MinIO Operator starts, it looks for TLS certificates and related secrets. If it doesn’t find any, then it will default to http over port 9090.

In order to create certificates, the MinIO Operator will create and submit a CSR. The service specifies the port (4222) for https, the secret name and additional details related to TLS. We can view this service

When we describe the secret, we can see that it contains the certificate and private key

If you wanted to use your own certificates for MinIO Operator and MinIO Console instead of having them generated for you, then you would create secrets operator-console-tls and operator-tls that contain the public certificate and the private key. The naming convention for tenant secrets is tenantname-tls.

In the step above, we used HTTP to connect to the MinIO Operator. If we want to enable TLS between your browser and the Operator Console in addition to communications between the MinIO Operator and MinIO Tenants, we would replace kubectl minio init with

This adds to the MinIO Operator deployment an environmental variable

When the MinIO Operator deployment starts, it launches pods for itself and its console, and submits the CSR, as seen by viewing its log:

When we view logs from the MinIO Console pod, we can see that we now have a port open for HTTPS.

To log in to the MinIO Operator and start managing MinIO, you must first port forward. We advise you not to expose the MinIO Operator to the public internet in order to protect your MinIO deployment.

Copy the JSON Web Token (JWT) that is displayed on your command line. Then, open a browser and navigate to http://localhost:9090. Paste the JWT into the login form and then click Login.

When the MinIO Console pod requires TLS, we can view the certificate in our browser after logging in. Click the lock in the address bar to view information about the certificate, which we can see was self-signed by Kubernetes.

Deploy and Secure a MinIO Tenant

We’re going to deploy a MinIO tenant inside the Kubernetes cluster, and for instructional purposes we’ll configure it for TLS and make it available to clients.

From your browser, within the MinIO Operator interface, click Create Tenant. For a detailed description of the tenant creation and configuration process, please see Simplifying Multi-Tenant Object Storage as a Service with Kubernetes and MinIO Operator.

Configure your tenant’s name, namespace, capacity and allocated resources. There are more options under the menus to the left, you’re currently in the Setup menu. When you’re ready, click on the Security menu.

You can see within the Security menu that TLS is turned on for this tenant, and the MinIO Operator is configured to automatically generate and update certificates in order to secure network traffic between nodes. Turning on AutoCert for the tenant will result in the same kind of certificate automation we saw above. This will submit a CSR to the Kubernetes API, create a certificate, mount that certificate into the appropriate pods of our stateful set, and our tenant is now protected by TLS and accessible via HTTPs.  

Managing certificates using the MinIO Operator saves time and reduces the risk of human error. Perhaps more importantly, you can assign certificates using whichever method fits into your existing process – the MinIO Operator GUI, in the tenant specification, or by submitting a CSR. Organizations with strict security requirements might choose to self-sign their certificates and simply submit the CSR. Those that expose MinIO to the public internet require certificates signed by a publicly trusted CA and then added to MinIO via the Operator GUI or the tenant specification. You might even choose a process that uses self-signed certificates internally and public certificates for external access. The choice is yours to make depending on existing security requirements, and whichever way you choose, the MinIO Operator handles the details for you.

What if you don’t want to use local self-signed certificates? You can add custom TLS key pairs for MinIO server and client, as well as a Custom CA certificate for each MinIO Tenant. If you already have certificates that were generated previously by your organization, this is where you would upload them. Or you can do what I did and follow these instructions to create certificates for localhost and then upload those.

There are a few requirements for custom certificates. The certificates you provide must include a FQDN for the cluster domain, as well as the MinIO Tenant name and namespace. The MinIO Operator sends data to and receives data from each MinIO Tenant and other services on your cluster, so you need certificates and private keys for server and client.  

If you maintain your own certificate authority (CA), then this requires a custom certificate as well. Otherwise, other services don’t know if they can trust MinIO’s certificates. You can attach as many CA certificates as required. Custom certificates, saved as secrets, for a MinIO Tenant are referenced in the tenant specification YAML.    

You can see all of our secrets to reassure you that they’re there

MinIO Tenants support Server Name Indication (SNI), enabling the MinIO server to provide the correct certificate for the hostname specified by the client. This is useful when you have a hostname mapped to a bucket and secured via TLS. The MinIO Operator will attach as many certificate keypairs to a MinIO Tenant as required. This workflow greatly simplifies deployment and ongoing management of certificates, freeing staff from such error-prone drudgery and enabling large scale operations.

In the example below, you can see that the custom certificate that I uploaded in the previous step

After you are finished configuring your custom certificates, click the YAML button on the top right to view the Tenant Specification and verify that they (and their secrets) are properly referenced. Then, click Create and the MinIO Operator will deploy your new tenant. After a few minutes, the tenant is up and running, its communications secured by TLS.

Renewing and Updating Certificates

Certificates are only issued for a certain period of time, so they need to be renewed before they expire. Failure to do so will prevent you from securing communications with TLS – and this may be the most common cause of certificate-related downtime. Tracking expiration dates and renewing certificates manually is time-consuming and error-prone.

When you use AutoGen for certificates, the MinIO Operator renews a certificate that is about to expire for you. When 80% of the time for which a certificate is valid passes, the MinIO Operator automatically renews the certificate using the Kubernetes API. For example, if we parse and decode the secret that holds the certificate for the MinIO Operator, we see a start and end date. These were included in the original API call that the MinIO Operator made to Kubernetes (the default is 1 year).

The MinIO Operator is about to be updated to simplify examining tenant certificate information. In the near future, describing a tenant will return a plethora of information, including certificate expiration. This will simplify troubleshooting so you can resolve certificate issues quickly.

You can also add certificates to your MinIO configuration even after it has been deployed. Take the case of a MinIO deployment that only serves internal traffic. Self-signed certificates were fine for internal users, but now you want to make a bucket available over the internet so you will need public certificates. We were using AutoGen for our certificates before, but now we will  upload the custom certificates, by using the MinIO Operator GUI, submitting a CSR or writing a new secret, and then restart the stateful set for that particular tenant using

If you’re running with custom certificates, the MinIO Operator GUI clearly shows you how much time is left on each certificate. The number of days remaining is color-coded based on urgency, and when a certificate expires, the expiration date is replaced by EXPIRED.

We’ve reached the end of our walkthrough of working with certificates using the MinIO Operator and the Kubernetes Cert Manager.

If you wanted to expose the tenant or console for access outside of the Kubernetes cluster, then a self-signed certificate won’t work. If this is the case, then you’ll want to read the next blog post because it will cover how to integrate the MinIO Operator, Kubernetes Cert Manager, certbot and Let’s Encrypt.

Conclusion

Security is an important consideration when deploying large-scale object storage systems and MinIO has benefited from the knowledge that only comes with hundreds of multi-petabyte deployments. MinIO encrypts data at rest and inflight. Encryption and decryption create minimal overhead for MinIO because of the attention we’ve paid to CPU instruction optimization.

You now have the tools to run secure multi-tenant, multi-cloud Kubernetes deployments to provide MinIO object storage to all your cloud-native applications.

If you have any questions, reach out to us on hello@min.io or join our Slack community.