Best Practices for Deploying MinIO in Virtualized Environments

Eco -

Virtualized data centers provide enterprises with software-defined infrastructure - servers, storage, network - so they can achieve the operational efficiency and flexibility of the cloud. Decoupling software from hardware gives IT the flexibility to operate multiple workloads on a shared pool of infrastructure resources. Virtualization gives enterprises the means to continuously innovate to meet business goals instead of devoting resources to maintaining rigidly-configured server farms. Virtualization allows for automation and orchestration, removing the burden of managing underlying infrastructure, speeding service delivery while reducing operational expense.    

Customers use MinIO in virtualized environments for a number of reasons. First and foremost, MinIO is used to bring cloud-native object storage to software-defined data centers. This is the primary use case for deploying MinIO object storage in virtualized environments. Our enterprise customers typically run VMware vSphere, but these tips apply to any hypervisor.

This blog post focuses on best practices for deploying MinIO in virtualized environments. These recommendations are meant to be a starting point and help you make sure that MinIO is installed into an environment where it is configured to deliver the expected performance and durability guarantees.

Deployment Guidelines

When deploying MinIO in virtualized environments, it’s important to make sure that the proper conditions are in place to get the most out of MinIO. MinIO is a high performance system, capable of aggregate speeds up to 1.32 Tbps PUT and 2.6 Tbps GET when deployed on a 32 node cluster. Make sure to adhere to your organization’s best practices for deploying high performance applications in a virtualized environment.

Avoid “noisy neighbor” problems. In order to get the most from MinIO, it should be the only workload running on the hypervisor, the same as you would do for any high performance system in a virtualized environment. This will prevent loss of performance caused by resources being consumed by workloads as they run on other VM’s.

Provision networking appropriately. Ensure there is (more than) enough Ethernet switch bandwidth to satisfy the requirements of applications you plan to run with MinIO. You can use a simple tool like mperf  to test.  You can use the comprehensive WARP benchmarking tool to see how your MinIO cluster will perform under various workloads.

Ensure availability. Each MinIO VM should run on its own dedicated hypervisor.  MinIO uses erasure coding to ensure availability and durability of data. You can see how MinIO handles loss of disks or nodes with erasure coding with this handy tool. Having multiple VM’s of a given cluster on the same hypervisor presents the risk of a single hypervisor taking down a larger than expected amount of the cluster which can affect the HA and durability guarantees MinIO offers.

Remove artificial bottlenecks. For best performance, MinIO requires local disks to back demanding workloads. Due to this, you should not run MinIO on top of other systems that provide their own durability. This is unnecessary because data saved in MinIO is erasure coded, and running other systems with their own durability methods underneath will create an artificial bottleneck. Examples of such systems include RAID, LVM or ZFS pools.  Do not run MinIO on top of a distributed file system such as NFS, GlusterFS, GPFS, etc.  Do not run MinIO on thin disks. The goal is to reduce complexity and potential bottlenecks, and maximize performance. For example, you can run MinIO on SAN disks, but this will add an extra layer of complexity and make it difficult to enforce performance requirements across shared storage.

Prepare the VM. If your hypervisor provides guest optimizations, make sure that the current version (e.g. VMware Tools) is installed to ensure you are taking full advantage of the virtual hardware’s capabilities.

Deployment Requirements

MinIO has lightweight requirements for CPU and RAM. The software runs on almost any hardware, making it well suited for a virtualized environment when properly configured.

The only limitation to MinIO’s performance is I/O, meaning that reads and writes can only be as fast as the underlying I/O systems, i.e. drive, controller or network, etc. Benchmarks have repeatedly demonstrated that MinIO is network bound, limited only by available bandwidth. Ideally, the workloads that run on MinIO should be network limited as well.    

For purely functional testing, MinIO can be run on hardware as light as a Raspberry Pi.  As such, even very minimally specced VM’s can work. This is helpful for people who want to learn and experiment with portable S3-compatible object storage, but not the right choice for production.

Production implementations place greater demands on underlying infrastructure. The requirements listed are a great starting point to ensure a successful deployment of a high performance production cluster. To achieve the best MinIO experience in virtualized environments, we recommend the following specs for testing (per VM):

  • 8 vCPU
  • 32GB RAM
  • 4 disks (backed by physical disks, as mentioned above). Although there is no technical limitation to running fewer disks, you will not be able to do robust testing of disk failure or erasure code settings, and you may not have sufficient throughput for very high volume workloads.
  • 10Gbps network. Please use the mperf tool listed above to ensure that VM’s are providing expected throughput. In some older hypervisors, artificial bandwidth constraints are put on the disks. Although it is possible to use less than a 10Gbps network, please bear in mind that even network-intensive small workloads quickly saturate a 1Gbps network

By using the WARP benchmarking tool mentioned above, you can determine whether you will need additional resources to satisfy your application's requirements.

High Performance Object Storage for Virtualized Environments

Lightweight yet scalable, resilient and performant, MinIO brings multi-tenant cloud-native object storage to virtualized environments. Make the most of your software-defined data center with the addition of the world’s fastest object storage.

Get started by downloading MinIO today. If you have any questions about getting the most out of MinIO in virtualized environments, ping us on hello@min.io or join the Slack community.