Benchmarking AIStor with WARP and Perf test

Modern applications such as AI/ML, advanced analytics and databases require high-performance object storage. AIStor’s combination of scalability and high-performance puts every workload, no matter how demanding, within reach. Our published benchmarks demonstrate that AIStor is the fastest object storage available, but it can only run as fast as the hardware and network you give it. Results from WARP and Perf Test can be used to track performance over time and proactively pinpoint potential problems before they grow into serious issues.  

WARP is an open-source full-featured S3 performance assessment software built to conduct tests between WARP clients and object storage hosts. WARP measures GET and PUT performance from multiple clients against a MinIO cluster. WARP has many options, configured by command line or via YAML files, allowing you to create tests that align with your workloads. Remember - it’s absolutely critical to assess performance in a meaningful way and test using data and a workload that represents the real world conditions in which your object storage operates.

In contrast, Perf Test is an easy-to-use automated performance test that provides a streamlined experience and concise results. We introduced Perf Test to provide AIStor administrators with the ability to run a distributed performance assessment of their MinIO clusters with a single command. Perf Test runs PUTS, then GETS, incrementally increasing load to pinpoint maximum aggregate throughput. Tests can take anywhere from a few seconds to several minutes to execute depending on your MinIO cluster.

There are some important differences between the WARP and Perf Test. WARP is a complete end-to-end tool that tests performance of your MinIO cluster via operations conducted by WARP clients. Client machines are conducting PUTS and GETS of objects across the network to your distributed MinIO cluster. This makes WARP an excellent mechanism for testing anticipated client-server performance. In contrast, Perf Test runs on MinIO instances to conduct PUTS and GETS of objects to other MinIO instances in the same cluster. As such, Perf Test tells you what the server side of the performance equation is capable of.    

WARP: A Complete Object Storage Benchmarking Tool

First, let’s take a look at WARP. WARP is a tool that allows you to thoroughly benchmark MinIO, or any S3-compatible object storage, performance under a variety of conditions. By default, WARP will perform a mixed test of read and write over all disks utilized by MinIO running in distributed mode.

For this demo, we’re going to be running WARP to test MinIO running in a distributed mode. 

WARP is located in a GitHub repository, where you will also find detailed documentation.We’re going to start by deploying MinIO in distributed mode:

Once MinIO is up and running you will be able to measure performance with WARP, so let’s get started.

Step 1: Create 4 more instances for WARP clients for a total of 8 instances.

Please follow steps from the previous blog post to deploy instances, but do not configure additional storage. The instances running WARP must be powerful enough to quickly generate objects and measure PUT and GET operations, but do not require the storage volume needed by a MinIO node.

Step 2: Install WARP on the clients and run it.

[root@ip-172-31-19-7]# wget https://github.com/minio/warp/releases/download/v0.5.5/warp_0.5.5_Linux_arm64.tar.gz
2024-08-08 20:06:04 (66.3 MB/s) - 'warp_0.5.5_Linux_arm64.tar.gz' saved [3943887/3943887]

[root@ip-172-31-19-7]# tar -xzvf warp_0.5.5_Linux_arm64.tar.gz
LICENSE
README.md
warp_logo.png
warp
[root@ip-172-31-19-7]# ./warp client
warp: Listening on :7761

Notice that each WARP client is listening on port 7761

Step 3: Run WARP and launch a mixed benchmark.

You’ve completed a basic mixed-mode performance test using WARP. While we used a mixed-mode test to demonstrate WARP which is more aligned to real-world usage. You can also break out individual operations (GET, PUT, STAT) into their own test runs instead of running a mixed-mode test.

We encourage you to refer to the documentation to learn about executing more test scenarios. For example, you can enable TLS and server-side encryption to measure their impact in your environment. You can stress infrastructure more by increasing the number of concurrent tests. You can use a random mix of object sizes, or specify an object size that matches your current environment and workload. You can configure tests to run for a defined period of time or to auto-terminate as we did above.

WARP results are shown on-screen and saved to a local CSV file compressed using zstandard, which can be used for further breakdown or analysis.

Perf Test for a Quick and Easy Performance Assessment

In the event that you don’t want the full flexibility of WARP, we’ve built a quick and easy way to measure internal MinIO performance using Perf Test. Perf Test is most commonly run in autotune mode, incrementally increasing load to pinpoint maximum aggregate throughput.

You can use mc to run Perf Test:

$ mc support perf object <alias>

Or, you can run Perf Test from MinIO Console, the browser-based GUI for configuring, managing and reporting on MinIO clusters. Simply navigate to the Support menu and click Performance, then click on the start button.

The performance test will run. During the test incoming traffic will be temporarily blocked. When it is complete, aggregate GET and PUT performance as well as performance of individual nodes is displayed.

For more details and examples for the Perf Test, please see this tutorial.

Warp Speed Ahead!

WARP contains just about every test (GET, PUT, DELETE, LIST, STAT, RETENTION) you might want to run against object storage, and test runs can include a custom mix of object sizes to match your workloads. Perf Test runs through set test parameters to pinpoint maximum throughput quickly. While WARP requires additional software, Perf Test is included with MinIO and can be run from the mc interface or MinIO Console. To aid in troubleshooting, customers can share Perf Test results with MinIO engineers with a single click.  

Get started by downloading MinIO today. Join our Slack community and exchange ideas with other MinIO users. If you have any specific questions about using WARP or Perf Test, please reach out to us on our Slack channel or send us an email at hello@minio.io.