How to Benchmark MinIO with WARP and Speedtest

Customers bring MinIO into their environments to gain high-performance S3-compatible object storage that runs on any physical or virtual hardware and Kubernetes. Our published benchmarks demonstrate that MinIO is the fastest object storage available, but it can only run as fast the hardware and network you give it.  Performance is a big focus for our team, so we do everything we can to help administrators get the most out of MinIO. In order to verify that underlying infrastructure can support MinIO optimally, we provide two methods to assess performance and identify potential bottlenecks: WARP and Speedtest.

Modern applications such as AI/ML, advanced analytics and databases require high-performance object storage. MinIO’s combination of scalability and high-performance puts every workload, no matter how demanding, within reach. When performance is critical, you need to know that you’re achieving the best performance possible. Results from WARP and Speedtest can be used to track performance over time and proactively pinpoint potential problems before they grow into serious issues.  

WARP is an open-source full-featured S3 performance assessment software built to conduct tests between WARP clients and object storage hosts. WARP measures GET and PUT performance from multiple clients against a MinIO cluster. WARP has many options, configured by command line or environmental variables, allowing you to create tests that align with your workloads. Remember - it’s absolutely critical to assess performance in a meaningful way and test using data and a workload that represents the real world conditions in which your object storage operates.

In contrast, Speedtest is an easy-to-use automated performance test that provides a streamlined experience and concise results. We introduced Speedtest in November to provide MinIO administrators with the ability to run a distributed performance assessment of their MinIO clusters with a single command. Speedtest runs PUTS, then GETS, incrementally increasing load to pinpoint maximum aggregate throughput. Tests can take anywhere from a few seconds to several minutes to execute depending on your MinIO cluster.

There are some important differences between WARP and Speedtest. WARP is a complete tool that tests performance of your MinIO cluster via operations conducted by WARP clients. Client machines are conducting PUTS and GETS of objects across the network to your distributed MinIO cluster. This makes WARP an excellent mechanism for testing anticipated client-server performance. In contrast, Speedtest runs on MinIO instances to conduct PUTS and GETS of objects to other MinIO instances in the same cluster. As such, Speedtest tells you what the server side of the performance equation is capable of.

WARP contains just about every test (GET, PUT, DELETE, LIST, STAT, RETENTION) you might want to run against object storage, and test runs can include a custom mix of object sizes to match your workloads. Speedtest runs through set test parameters to pinpoint maximum throughput quickly. While WARP requires additional software, Speedtest is included with MinIO and can be run from the mc interface or MinIO Console. To aid in troubleshooting, customers can share Speedtest results with MinIO engineers with a single click.      

WARP: A Complete Object Storage Benchmarking Tool

First, let’s take a look at WARP.  WARP is a tool that allows you to thoroughly benchmark MinIO, or any S3-compatible object storage, performance under a variety of conditions.  By default, WARP will perform a mixed test of read and write over all disks utilized by MinIO running in distributed mode.

For this demo. we’re going to be running WARP to test MinIO running on AWS in distributed mode. We’ve been doing this pretty often lately, so we can advise potential and current customers of the highest-performing configurations we have found by testing them.

WARP is located in a GitHub repository, where you will also find detailed documentation.

We’re going to start by deploying MinIO in distributed mode. You can download MinIO and install it in your environment by yourself, or you can follow this tutorial to install MinIO on AWS EC2.  

Either way, we are going to start with a distributed MinIO deployment:

Once MinIO is up and running you will be able to measure performance with WARP, so let’s get started.

Step 0: Create 4 instances (also known as nodes) with Distributed MinIO.

Please read and follow steps provided in How to Install MinIO in Distributed Mode on AWS EC2.

Step 1: Create 4 more instances for WARP clients for a total of 8 AWS instances.

For all Amazon UI or AWS CLI, please follow steps from the previous blog post to deploy instances, but do not configure additional storage. The instances running WARP must be powerful enough to quickly generate objects and measure PUT and GET operations, but do not require the storage volume needed by a MinIO node.

Step 2: Add hosts in /etc/hosts file at WARP clients.

1. Get the IP Addresses of the MinIO instances
2. Get the IP Addresses of the WARP clients
3. Add those address in the /etc/hosts file:
   

Notice that we recommend listing and working with hosts in consecutive order. This simplifies launching WARP from the command line and makes it easier to locate each WARP client and MinIO host.

Step 3: Install WARP on the clients and run it.

[root@ip-172-31-19-7 ec2-user]# wget https://github.com/minio/warp/releases/download/v0.5.5/warp_0.5.5_Linux_arm64.tar.gz
2022-02-08 20:06:04 (66.3 MB/s) - 'warp_0.5.5_Linux_arm64.tar.gz' saved [3943887/3943887]

[root@ip-172-31-19-7 ec2-user]# tar -xzvf warp_0.5.5_Linux_arm64.tar.gz
LICENSE
README.md
warp_logo.png
warp
[root@ip-172-31-19-7 ec2-user]# ./warp client
warp: Listening on :7761

Notice that each WARP client is listening on port 7761

Step 4: Run WARP and launch a mixed benchmark.

You’ve completed a basic mixed-mode performance test using WARP. While we used a mixed-mode test to demonstrate WARP, you will typically obtain better results that are more aligned to real-world usage) by breaking out individual operations (GET, PUT, LIST) into their own test runs instead of running a mixed-mode test.

We encourage you to refer to the documentation to learn about executing more test scenarios. For example, you can enable TLS and server-side encryption to measure their impact in your environment. You can stress infrastructure more by increasing the number of concurrent tests. You can use a random mix of object sizes, or specify an object size that matches your current environment and workload. You can configure tests to run for a defined period of time or to auto-terminate as we did above.

WARP results are shown on-screen and saved to a local CSV file compressed using zstandard.

Speedtest for a Quick and Easy Performance Assessment

In the event that you don’t want the full flexibility of WARP, we’ve built a quick and easy way to measure MinIO performance using Speedtest. Speedtest is most commonly run in autotune mode, incrementally increasing load to pinpoint maximum aggregate throughput.

You can use mc to run Speedtest:

$ mc support perf object <alias>

Or, you can run Speedtest from MinIO Console, the browser-based GUI for configuring, managing and reporting on MinIO clusters. Simply navigate to the Support menu and click Performance, then click on the start button.

The performance test will run. When it is complete, aggregate GET and PUT performance as well as performance of individual nodes is displayed.

For more details and examples for Speedtest, please see this tutorial.

Warp Speed Ahead!

Get started by downloading MinIO today. Join our Slack community and exchange ideas with other MinIO users. If you have any specific questions about using WARP, please reach out to us on our Slack channel (Matt Sarrel and Cesar Celis) or send us an email at hello@minio.io.