Relaxing the 32-Node Restriction on MinIO

MinIO previously imposed a much-discussed 32-node per cluster limit, which we lifted in December. There was no hard technical reason for the 32-node limit — you can use MinIO to create 1,000 node clusters. That reason we imposed the limit was to drive good architectural decisions. We learned, however, that some saw this as a limitation (indeed some competitors suggested that 32 nodes was ALL we could scale to) and so we lifted it. We understand that part of using open source software is fiddling around with as many moving parts as possible. Many users want as much flexibility as possible, even the flexibility to do something we do not recommend.

Nonetheless, the 32-node cluster has some material benefits from an architecture perspective and we wanted to use this post to point them out. The timing is right, it doesn’t look like we are defending the older limits since it does not exist anymore.

The reason we had the limit in the first place was to encourage good architecture design. Just because building 1,000 node clusters is possible doesn’t mean it’s a good idea - especially when paired with Kubernetes. MinIO is often used by developers who might not have experience creating or managing storage architecture. Our goal with the 32 node limit was to create guardrails so that users weren’t unknowingly creating storage clusters that would be impossible to support long-term.

Why Just 32 Nodes?

The main challenge for clusters with more than 32 nodes is supportability. With large failure domains, it becomes more difficult to do maintenance activities, more difficult to recover from a failure and next to impossible to successfully back up data. Further, with 32 nodes per cluster you can store a LOT of data. A 32-node cluster is capable of storing up to 200 petabytes. If a user needs more than 200 petabytes of data to act as if it were in the same cluster, it is simple to create multiple clusters and expand your existing deployment, each with the appropriate number of nodes (again we like 32)  providing the same functionality without violating sizing best practices.

The complexity that comes with managing over 200 petabytes on a single cluster begins to get daunting. When you have a very large fault domain, if a few servers go down the entire infrastructure can go down. Failures are a support nightmare and are more likely to lead to data loss. Let’s be clear - failures are a ‘when’ not ‘if’ issue, especially when dealing with large numbers of nodes. Smaller node configurations make manageability and root cause analysis easier.

Another problem with single large fault domains is the time required to back up or move data. Data does not exist in a vacuum — there are also networking issues to consider. The larger the failure domain, the lower the likelihood that a company will have the network capacity to effectively back up or move data if it’s stored in extremely large chunks.

How Cloud Native Infrastructure works

The basic premise of cloud native architecture, whether it is for compute or storage, is to consume resources in small, discrete chunks. As you scale, you add additional instances and pack them as densely as possible in your infrastructure instead of scaling one instance up infinitely.

Even in the public cloud, there are physical limits to what infrastructure can handle (# of buckets, # of API requests per hour etc). Ignoring those limits is a recipe for data loss and/or downtime.

While there is nothing magical about 32 nodes, we have found it to be a good proxy for balancing failure domain and scale. Ignoring that recommendation starts to contradict the basic premise of running containerized workloads: that applications (and their associated storage) should be broken up into pieces that are as small as possible.

MinIO’s Sizing Recommendations

MinIO’s sizing recommendations haven’t changed since the limit restrictions were removed. Clusters should ideally have at least 4 nodes, to meet redundancy requirements, and a maximum of 32 nodes. If you need additional storage capacity, the best practice is to deploy new instances of MinIO as you scale. Even if each cluster has only 32 nodes, the clusters can be federated together, allowing them to operate as one giant cluster (and scale to infinity) while still keeping the architecture manageable.

This architecture is less complex to manage at scale than an enormous cluster but still provides the potential for limitless scaling.

Relaxing the 32 Node Restrictions

Our growth has created an unbelievable number of configurations and use cases and that has led some in the community to ask for the relaxation of the 32 node restriction to support their specific deployments. In addition, it was serving as a distraction — often from people who didn’t understand failure domain best practices.

One of the beauties of open source software is that people are free to play around and learn it the hard way — so, if you feel strongly that running a 1,000-node storage cluster is right for your production application, you are welcome to try it out. You won’t likely see any problems at first, but when you try to back up, move data or managing maintenance - well... we did tell you so.

Want to start experiment with MinIO? Download it now.