How Server Pools Streamline Storage Operations

MinIO takes particular pride in the concept of manageability. This is born out of our relentless focus on simplicity. Our belief is that a single individual with moderate technical capability should be able to manage a multi-petabyte instance of MinIO in their free time. To make this a reality, MinIO critically evaluates the “traditional” way of doing things and dispenses with outdated or overly complex approaches.

One outcome of that thinking is the concept of server pools. Server pools allow MinIO to scale, shrink and otherwise be maintainable in the face of heterogeneity - particularly hardware heterogeneity which is a constant challenge for long lived, distributed systems.  

A server pool is a set of minio server nodes which pool their drives and resources, creating a unit of expansion. All nodes in a server pool share their hardware resources in an isolated namespace.

The server pool concept is essential for operational efficiency at scale because it deals with the entire storage lifecycle, including planning, deployment, expansion, upgrades and more. In short, the ability to treat an entire hardware cluster as a single resource goes a long way towards setting you free from managing the minutiae typically associated with enterprise storage.

Server pools enable customers to focus on making the most out of their object storage solution instead of framing the issue around which hardware to buy. Removing the traditional one-track and now short-sighted approach of starting each year by writing the trusted storage appliance vendor a big check to add more drive space sets IT and DevOps teams free to leverage the architectures and methodologies that are driving software innovation across all industries.

Streamlining Operations

A MinIO cluster is built on server pools, and server pools are built on erasure sets, so let’s start there.  

Erasure sets, built per server pool, are sets of nodes and drives to which MinIO applies erasure coding to protect data from loss and corruption. Erasure coding breaks objects into data and parity blocks and can use these blocks to reconstruct missing or corrupted blocks if necessary. With MinIO’s highest level of protection (8 parity or EC:8), you may lose up to half of the total drives and still recover data.

The erasure set configuration is the commonality between all objects written to a server pool. Erasure code stripe size and parity settings are configured when the server pool is brought into service. Erasure coding is a core mechanism in MinIO that dictates how hardware nodes and drives are used to store and retrieve objects. MinIO deployments begin by planning erasure sets and their server pools. We’ve even got a handy erasure code calculator to help you get started choosing the right combination of servers, drives, erasure code stripe size and parity to meet your needs for fault tolerance, capacity and performance.    

Server pools make deployment and planning for expansion easier. As mentioned previously, MinIO can be deployed on whatever hardware (or cloud instances) are available or needed at the time of provisioning. MinIO deployments can grow at your pace and on your preferred hardware. Hardware in server pools must match, but other than that you are free to run a server pool of anything, and then add that entire server pool to MinIO with a single command.  

MinIO runs on virtually any COTS hardware. For a variety of reasons, customers are able to run those servers and drives far longer than traditional appliances. This means six to ten years vs. the old three to five. Longer HW lifespans result in significantly more HW heterogeneity - even if the customer only has one HW provider. Most object stores are incapable of dealing with heterogeneity, much less significant amounts.

Storage pools are the mechanism for attacking this problem.

By grouping homogeneous HW profiles into server pools, MinIO can group them into a performance optimized namespace.

This is an important point. Yes, you will still have to “wait” for the HW stragglers in your group, but not every group will have stragglers. Further, you can clearly identify which HW profile/storage pools are the slowest using SUBNET Health. Those are the prioritized target(s) for decommissioning or tiering.  This allows an Operator to upgrade in place without disruption.

The operational flexibility of this approach cannot be understated. It actually puts the enterprise on better footing than even the hyper-scalers because the procurement teams can buy from anyone, including the hyperscalers (due to MinIO’s tiering capabilities) where they simply view them as another commodity HW option.

The other important point here involves rebalance-free, non-disruptive expansion. We have written about this at length here, but the TL;DR is that rebalancing is problematic in distributed systems and the common antidote (excess capacity) is a poor option. With MinIO’s server pool approach - rebalancing is not required to expand.

Server Pools for Operational Efficiency

MinIO provides the tools that enterprise storage admins need for data availability, continuous data protection, and identity and access management, in a server pool based architecture that streamlines expansion and automates operations.

The server pool design streamlines operations and enables operational efficiency because instead of getting bogged down managing individual nodes, admins manage groups of MinIO servers as a single entity.

Download MinIO and start building your object storage cloud. It’s straightforward, software defined for greatest flexibility, and S3 API compatible so it’s ready for your workloads. Any questions? Join our Slack Channel or drop us a note at hello@min.io.