Building a Moat Around an Object Store

Most developers, engineers, architects and DevOps folks know MinIO. Not all know that the only thing we do is software-defined object storage. We don’t do file or block. We don’t offer a service, it is self-hosted.

Our focus is singular.

The result is that our object store is objectively, based on adoption, awards and customer feedback the best in the world. We do nearly 1M Docker Pulls a day. We have won every analyst award and Datanami’s Reader’s Choice Award. We have won Gartner’s Peer Insights Customer Choice Award.

But there is another side to our singular focus on object storage and that is all of the peripheral things that we do to enhance the developer experience that don’t ever get graded by analysts or called out in reader surveys.

Collectively these efforts, a handful of which are outlined below, create a massive feature moat around the core object store. As a result, the distance between MinIO and the rest of the industry, particularly those vendors who do object storage as a side gig, continues to grow.

These “peripheral” components are key to an exceptional developer experience because they enhance what the developer can do, the time in which the developer can do it and the maintainability of the developed architecture/infrastructure.

In many cases, if this work is not implemented correctly, the application will not operate correctly. That requires troubleshooting, connectors, drivers and the like. If the developer who wrote the code leaves, well, that introduces another set of ever compounding problems.

Here are a few examples:

OS Tools and SDKs - When we first started developing MinIO as a drop-in replacement for AWS, we obviously spent a lot of time with the AWS SDKs. Unfortunately, they were an afterthought for AWS and it showed in the quality of the documentation, its accessibility and its completeness. We recognized that this wasn’t just going to limit the growth rate of AWS, it was going to limit the growth rate of MinIO as well.

So we decided to make the investment to write our own SDKs. We were building a server of the finest craftsmanship and we needed SDK’s and API’s that matched that level of craftsmanship.

You can find our work here. It includes SDKs for .NET, Golang, Haskell, Java, JavaScript, and Python. Rust is in development.  

These SDKs, because they were S3 compatible and superior to those provided by AWS, became the standard in the industry - even for those developers working exclusively within AWS’ S3 service.

We extended that craftsmanship to our OS tools as well. Most notable is the unix-like mc interface but we also have tooling for Microsoft as well.

The end product was a collection of core documentation and tools that developers could use with confidence. This was the equivalent of turning the nitrous lever for the acceleration of S3 as a protocol. We continue to invest heavily here and will do so until there is no requirement.

Identity and Access Management - There are many LDAP implementations. Active Directory is certainly the most popular, but there is OpenLDAP, IBM Tivoli and Apache Directory Server to name a few. There is also OpenID - which also has dozens of implementations.

MinIO invested in its object store such that all of them would work - in the cloud or on-prem. Many of those solutions don’t speak the S3 API for IAM or even the STS tokens. MinIO’s IAM is built with AWS Identity and Access Management (IAM) compatibility at its core and presents that framework to applications and users no matter the environment - providing the same functionality across varying public clouds, private clouds and the edge.

MinIO extends AWS IAM compatibility with support for popular external identity providers such as ActiveDirectory/LDAP, Okta and Keycloak, allowing administrators to offload identity management to their organization's preferred SSO solution.

This is a critical theme that will repeat - MinIO connects to the outside world but has enabling technology that is bespoke to MinIO and is designed for the requirements of a high-performance object store. You can absolutely use the most widely deployed options from the market, but we are committed to building that best of breed directly into the product.

Further, MinIO provides multiple ways to achieve this - again, targeting the developer and catering to their preferences. For example, MinIO’s IAM is built to support both manual (static) and programmatic (dynamic) application-focused identity management. For manual identity creation, customers can use the MinIO Console. For programmatic identity creation, customers can use MinIO’s `mc` command line interface.

It is always the little things that matter. Case in point, MinIO recommends creating one user per application to mitigate the security risk associated with leaked long-lived user credentials. As a result, MinIO does not limit the number of users that can be managed and the number of users has negligible impact on system performance.

The list goes on and we highly recommend our Identity and Access Management page to truly understand the depth of the effort.

Key Management and Key Encryption - As object storage grew in popularity and in the data stored on it, certain challenges were encountered. One of those challenges related to key encryption. Traditional Key Management Services (KMS) were not equipped to handle billions of keys. When doing per object encryption - this was the challenge.

While MinIO supports every major KMS, we also developed a native option for key encryption. MinIO’s Key Encryption Service (KES) is a stateless and distributed key-management system for high-performance applications. It is designed to be run inside Kubernetes and distribute cryptographic keys to applications. It is required for MinIO Server-Side Object Encryption (SSE-S3).

KES works as an intermediary between the MinIO cluster and the external KMS, generating encryption keys and performing encryption operations as needed and unconstrained by the limitations of the KMS. As a result, there is still one central KMS protecting master keys and acting as the root of trust within the infrastructure but from the developer’s perspective. KES simplifies deployment and management by removing the need to spin up a KMS per set of applications. Instead, an application can request a data encryption key (DEK) from a KES server or ask the KES server to decrypt an encrypted DEK.

We are expanding KES to be its own full blown KMS so watch this space. Again, the developer has choices, MinIO never locks you in, it just provides optionality.

Load Balancing - Another area where the modern application stack can prove problematic. In the modern data processing environment, it is common to have 100s to 1000s of nodes pounding on the storage servers concurrently. Traditional load balancers, built for serving web applications across the Internet, are at a disadvantage here since they use old school DNS round-robin techniques for load balancing and failover. These DNS based solutions just don’t scale well in the face of modern requirements.

MinIO works with a wide variety of software-defined load balancers like NGINX, HAProxy, and Envoy Proxy. These are full-featured and handle complex web application requirements but are not built for high-performance, data-intensive workloads.

Modern data processing environments move terabytes of data between the compute and storage nodes on each run. While load balancers are not supposed to impair performance, they often do in these environments because they have limited aggregate bandwidth and introduce an extra network hop.

MinIO developed its Sidekick load balancer to solve the network bottleneck by taking a sidecar approach instead. It runs as a tiny sidecar process alongside each of the client applications. This way, the applications can communicate directly with the servers without an extra physical hop. Since each of the clients run their own sidekick in a shared-nothing model, you can scale the load balancer to any number of clients.

In a cloud-native environment like Kubernetes, Sidekick runs as a sidecar container. It is fairly easy to add Sidekick to your existing applications without any modification to your application binary or container image.

Again, these are options that we created for developers because we are developers and know the power of having a well-thought-out, all batteries included product.

Building a Moat

While improving the developer experience is a primary driver of our relentless efforts to build the world’s best object store, there are other implications too.

First, every time we make one of these additions we illustrate the point that the object store itself is just the start of a great product experience. Our craftsmanship there is well documented (from our erasure coding implementation to our encryption) but as a data store we see the world more holistically and that manifests itself in these extensions.

Second, each and every additional feature or extension further differentiates MinIO from those vendors who are simply looking to “check the box” on object storage. There are too many of these. Their primary product is SAN/NAS. That market is not just competitive, it is shrinking. So these companies have hopped on the object storage bandwagon - but in truth, their heart isn’t in it and it shows in their product offerings. They may make a first class SAN/NAS but then layer on a third class object store. They may tell you that the “unified” offering is best in class – but developers know the truth, even if the old school IT or finance folks are unable to make the distinction.

Early in my career I worked in the State Department for Secretary of State James Baker. He said something once that I remember to this day. It went something like this: “When you mix ice cream with manure, it still tastes like manure.” His wording might have been a little more descriptive - but the point is still there. If you mix good with bad you get bad.

Each bit of polish we put on our object store - from KES KMS to our SDK, builds a deeper and wider moat around our product, making it harder and harder for others to compete. Each feature itself is a “bridge too far” for them.

Collectively these just create two classes of object store – MinIO and everyone else. We hope you appreciate the difference.

As always, we welcome your thoughts. Join us on Slack or drop us a note on hello@min.io. Conversely, you can always sign up for the newsletter to make sure these types of posts find their way into your inbox on a monthly basis.