Secure Multi-tenant Object Storage for Internal and External Apps

Secure Multi-tenant Object Storage for Internal and External Apps

A common cloud-native architecture is to store data in object storage, such as MinIO, and make it available to a variety of applications, both internal and external. Data must be protected from inspection and tampering as it travels across networks, especially untrusted external networks. The standard for this is encryption using Transport Layer Security (TLS) v1.2+. MinIO Operator provides the ability to generate and allocate TLS certificates as part of the tenant deployment process. MinIO supports the use of multiple TLS certificates for each tenant, where each certificate corresponds to a specific domain name. MinIO uses Server Name Indication (SNI) to provide the right certificate for a given request.

Since Day One, MinIO has included encryption functionality to secure  data  at rest and in transit. When  sensitive data over the wire, such as STS credentials, passwords or the objects themself, you can have peace of mind knowing they will be delivered to MinIO in a confidential and secure way.

In this post you will learn how to:

  • Install MinIO Operator
  • Deploy a MinIO tenant
  • Add external TLS certificates to tenants
  • Access MinIO tenants using different domains (SNI support)

Installing MinIO Operator

In your Kubernetes cluster install MinIO Operator via krew:

kubectl krew install minio

Kubectl minio init

Alternatively, you can also install the operator manually:

kubectl apply -k github.com/minio/operator/\?ref\=v4.4.13

Login to Operator

If you installed operator via krew, then run:

kubectl minio proxy

Copy the JWT (JSON Web Token), open a browser and go to http://localhost:9090

If you manually installed operator, then you need to expose the operator service using the kubectl port-foward command, and then manually extract the authentication token:

kubectl port-forward svc/console -n minio-operator 9090

kubectl -n minio-operator  get secret $(kubectl -n minio-operator get serviceaccount console-sa -o jsonpath="{.secrets[0].name}") -o jsonpath="{.data.token}" | base64 --decode

Copy the JWT to the login form and press Login.

Deploy a new MinIO tenant

Before we start, let’s take a moment to describe our overall solution. Suppose you have the following requirement:

To accomplish this, you will need a MinIO tenant deployed inside the Kubernetes cluster that is accessible by both internal and external clients, and not only that, they will use different domains. Let’s do it.

From your browser, in the MinIO Operator, click the Create Tenant button. You will be presented with the following screen.

You can configure your tenant with details such as name, namespace, capacity, number of nodes, etc. You can explore the sub menus that contain additional options to tweak your tenant. Click on the Security menu.

In the Security menu, you can see MinIO Operator will automatically generate certificates to secure internode communications, but what about your internal and external domain certificates?

Click on custom certificates and you will see additional options to upload your own certificates. These are usually a public.crt and private.key file.

You can upload as many certificate keypairs as you need  - and thanks to SNI support MinIO will know the right certificate to return for each client.

When you are ready click Create.

After a few minutes your tenant will be online. Return to the Security tab and you will see your custom certificates are configured. From this screen, you can add additional certificates and remove or rotate existing ones if needed.

Now  you need to expose your Kubernetes service. Use your preferred method: NodePort, LoadBalancer or Ingress controller. For reference, we have a tutorial about using NGINX with MinIO.

Verify the TLS Certificates

Once the tenant is exposed to internal and external clients, it is very easy to verify that  SNI is working properly using openssl:

$ openssl s_client -connect external.corporate.domain.com:9000 -showcerts
...
...
-----BEGIN CERTIFICATE-----

MIIERTCCAq2gAwIBAgIRAOYlnnAjbFXVPszN+sup5qcwDQYJKoZIhvcNAQELBQAw

dzEeMBwGA1UEChMVbWtjZXJ0IGRldmVsb3BtZW50IENBMSYwJAYDVQQLDB1hbGV2

c2tAYWVyaXRoIChMZW5pbiBBbGV2c2tpKTEtMCsGA1UEAwwkbWtjZXJ0IGFsZXZz

a0BhZXJpdGggKExlbmluIEFsZXZza2kpMB4XDTIyMDMyOTAzNTA1NloXDTI0MDYy

OTAzNTA1NlowUTEnMCUGA1UEChMebWtjZXJ0IGRldmVsb3BtZW50IGNlcnRpZmlj

YXRlMSYwJAYDVQQLDB1hbGV2c2tAYWVyaXRoIChMZW5pbiBBbGV2c2tpKTCCASIw

DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM+tUEbdvMXK5u+G6TRFVSJFKBVd

m16c9ISah7K+zXTgWDjMCVRJyI7EAcTM5S/pcEMbhKq78aEG6rky7oWmuU/d5RmZ

4kY2r02lH1QlmpIpxSoeRdg49dHBnKb/bVw6yLUODrmJmAZHIccgo8dHRN2fyaVO

KTABFDWRe5ySWIwjh1316Zh1STpRxAZZF7itA800VMcSTRgYvR2+l73FwVe71zhc

51lheYzyZc6Q9KPWraN/dzvjnakRJ+zo6LUDHoYRuxXSH4LcZUJHF8MJmQF9b29i

rCQWNkJNAFdgE9CwMh6E36Xs9L8+g/h5mrZXywbTymCv13BBCPwlpDWriqkCAwEA

AaNyMHAwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMB8GA1Ud

IwQYMBaAFGcBDY05hBWXbpbB/B7ecLxUqB9EMCgGA1UdEQQhMB+CHWV4dGVybmFs

LmNvcnBvcmF0ZS5kb21haW4uY29tMA0GCSqGSIb3DQEBCwUAA4IBgQBf3CnBkPPZ

G8jSfWBcnvMx/MZ2rQe+k3HlEkspX6eUH2H0pUI7b0FzLOuV0n4eRYQ8o8MeeStq

jYG6qS9YI38+YpZ49U28+ZUJ286VOSSWvjgzug1DzTUmuHN2rXVYpIqHXlECE54g

hsLaLOBlbtQLJzeuECb3N2YRtqTDD19wvIPYjYuwL+kHsj2x2BhJXIHtporxrouo

AD1CxxTjGtIlYb5E4vKHTAjNud6ODn637m09DXMheJIpcY1WmY9Ke3FxRwZOAai5

zv2ZutMpuCpMfSYqZtWYcYEbYhs5r5yuDMVWaiI5NngVpOd7OghZ9kmi7fbwbhdC

eK2TeTyaXvb+tsghf+L9WCpRbNjLR6q3JKip0GHxPq0hB0YpZV6olaWm/DiTs7S3

ry76rhguoZsfzeGTpaIvQd0G/akBcNKWjDTAlUyevoi9HYRviGSWAaNGW8yn+NMl

PDZH6VbrDcY0dlZkVTlHt4mel3q1GhgXlGM5go5un8K8bh8afUFEwL8=

-----END CERTIFICATE-----



$ openssl s_client -connect internal.corporate.domain.com:9000 -showcerts
...
...
-----BEGIN CERTIFICATE-----

MIIERDCCAqygAwIBAgIQJhMrQ72DRT0tnW5PQupkWTANBgkqhkiG9w0BAQsFADB3

MR4wHAYDVQQKExVta2NlcnQgZGV2ZWxvcG1lbnQgQ0ExJjAkBgNVBAsMHWFsZXZz

a0BhZXJpdGggKExlbmluIEFsZXZza2kpMS0wKwYDVQQDDCRta2NlcnQgYWxldnNr

QGFlcml0aCAoTGVuaW4gQWxldnNraSkwHhcNMjIwMzI5MDM1MTA2WhcNMjQwNjI5

MDM1MTA2WjBRMScwJQYDVQQKEx5ta2NlcnQgZGV2ZWxvcG1lbnQgY2VydGlmaWNh

dGUxJjAkBgNVBAsMHWFsZXZza0BhZXJpdGggKExlbmluIEFsZXZza2kpMIIBIjAN

BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwN0GRTnYyxi3mzFDFBUaYt56tVe4

J6DBzKj99IxJualzvu81rnoHoyZVj5NCxTKTko/rfG6Bcz5dyCSQJ8tZJnhNOHwk

v40i0xs+x41AE2v5P4tAD4ouiBceDQvUbilEM6gfQANXTLb3p1HzIVKzxaqWWn5V

xumPWlnhnPBnt4HosKW5ett4w2+hG4X4w5sE23VLkIceQjCjI4Ym+tzXSjf81Uxk

j7cj4dyEdZnnesQehS0DRhiBOsQz+pl0SMzVYby1Jlk1utpwz5XkQLQ96CfRLvn/

X0RxgnmL/pbqAYrN8WuFwn02e1bluL/vkTew8X8YldGm2lDRlNGiTnX1HQIDAQAB

o3IwcDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwHwYDVR0j

BBgwFoAUZwENjTmEFZdulsH8Ht5wvFSoH0QwKAYDVR0RBCEwH4IdaW50ZXJuYWwu

Y29ycG9yYXRlLmRvbWFpbi5jb20wDQYJKoZIhvcNAQELBQADggGBAFotl1RzbHhH

359vl2eMkYztnsx+L0BxfXeS+5nPInMQBC0/iCo2ZH3GZHTLFjR4QwtdbDYlXwm0

vbNFBWdwTkzn3EP4wsbID2Xd6LbWsU5gHv1ErUARZhI21+cUWiInn0hSU4DSWY3v

EPbdYg2eAZNUBVvnsAMDJDjGz0xS5qh7rtqBQcb3mC5Svc1+qDZPmtDtdSDc28rW

Pp0K+eV0oGHliKVhpCgk0NBRbYy4vUqt+NFP3s3FjRwKWl2Jybz4MUT0B4wO5sko

GdXabKrlogtAU6Gz5Q0puoMCi0Tm8mTjxWO8r4ATWy3Twpqeok4VaPLyxi2Rcnv6

mGNwPH0dwrIbGubMzfmu6pkaAv0uIyULzPE6NfjJN06XnmbKwbnvHpzwz7np4C0q

VGb9n6vxNoWdxI3Cyt+4qwM8s+4wCjg8eQEie3wr9mVWn2KP2cdUZ6QMaPnUmXUf

f2p1yUTjVkYonxQRygsiyw1woloPrYfXcyIfKIekXdrFWoyLIsdHJQ==

-----END CERTIFICATE-----


Copy the encoded certificates and go to https://www.sslchecker.com/certdecoder to confirm your domains are correct. You can decode a certificate with openssl using openssl x509 -text -in filename.

Important Notes

Configuring multiple certificates, for example internal and external, for MinIO tenants may appear cumbersome, but the effort is far outweighed by the benefit.  When certificates expire, you will have to rotate two certificates instead of only one. However, from a security perspective it’s better to separate your internal and external domain certificates because the information on them is public and you’ve just seen how easy they are to decode!

TLS certificates are a good source of information for attackers when planning and conducting reconnaissance. The information they contain may divulge details of  your internal infrastructure. Careful planning on your part is needed to find the right balance between security and functionality when designing your multi-tenant certificate strategy.

Conclusion

MinIO encrypts data when stored on disk and when in-transit. CPU instruction level optimizations enable encryption and decryption to take place with negligible performance overhead. MinIO customers leverage secure multi-tenant Kubernetes deployments to make S3 compatible object storage available to cloud-native applications for programs such as data lake analytics and streaming AI/ML. Multi-tenancy increases hardware efficiency and decreases cost. MinIO secures each tenant separately, and encrypts data saved on drives and transmitted across the network. MinIO is fully compatible with S3 encryption semantics so you know your application will function seamlessly and securely. MinIO includes support for other key management services such as Hashicorp Vault and Gemalto KeySecure.

If you have any questions, ping us on hello@min.io or join the Slack community.