Supermicro GrandTwin™ SuperServers and MinIO for Dense Rack-Optimized Storage

We recently had the opportunity to performance test MinIO on the new Supermicro GrandTwin™ SuperServer SYS-211GT-HNTR 2U server enclosure deployed in a quad server configuration, meaning that each chassis holds four server nodes. Each server, or node, can accommodate a single 4th Gen Intel Xeon processor, up to 4TB of RAM and up to 6 hot-swappable NVMe/SATA drives. We tested with an Intel Xeon CPU, 512GB of DDR5 RAM, 2 100GbE NICs and six Kioxia CD6-R 7.68TB NVMe drives. These are the first servers we’ve evaluated that rely on Intel’s Sapphire Rapids chipset. We were impressed by the overall performance of this dense rack-optimized server platform, and we'll continue to publish results and analyses of MinIO performance on that cluster.

Today, we're going to dig into small file performance on this cluster. Small file performance is important because database vendors are increasingly moving away from SAN block storage to object storage. Object storage use cases are expanding beyond traditional large file use cases such as archive and backup. Object stores are capable of handling petascale database tables that are stored as smaller table segments or extents and require ACID properties. For this reason, today’s object storage must be versatile to support workloads that use files that are less than one MB in size. Small objects create performance challenges for storage systems, particularly in the areas of concurrency and metadata management. For more on small files, please see Small Objects and their Impact on Storage Systems.

Where do massive numbers of small files (as in tens of PB) come from and what are they used for? New AI-driven applications are trained with massive amounts of small data files for inferencing, retraining and other aspects of model development. Analytics is another use case that is built on top of a data lake of small files. IoT applications draw on readings from various devices and sensors, and these readings typically come in as very small streaming messages. One final example is log file analytics that has shifted away from batch processing and analysis of large files to analyzing individual events in real time.

MinIO leverages server hardware to the fullest capacity. The result is that, as our testing has verified time and time again, MinIO deployments are typically bottlenecked by the network. In this testing, even though we had separate 100GbE networks for external access to MinIO and for internode communications, we were bottlenecked by the 100GbE networks. For this reason, we recommend that you rely on 100GbE as a minimum configuration, if not 200GbE or even 400GbE to truly achieve the mix of performance and scale that distinguishes MinIO. We’re currently planning tests for a MinIO deployment on a 400GbE network and are really looking forward to sharing results with you in a future blog post.  

Object storage benchmarks have traditionally been presented in throughput. Throughput was, and is, considered the best measure of object storage performance because it measures the system’s ability to read and write large volumes of unstructured data. However, as object storage technology has evolved, so have use cases and the metrics reported. MinIO handles modern object storage workloads (many small files with high concurrence) with the same ease with which it handles traditional high-throughput use cases such as backup and archive.

In the world of small files, objects per second is a more applicable measure of performance than throughput. The reason is that it measures the system’s ability to read and write many files concurrently - thereby providing a better approximation of performance for big data analytics use cases.

Small Files Benchmark

The dense rack-optimized Supermicro GrandTwin™ SuperServer really distinguished itself from other commodity hardware when we ran our small files benchmark. Our test cluster delivered first-class performance, and that is why our customers love running MinIO on Supermicro hardware in their datacenters, colocation facilities and at the edge to back resource intensive workloads such as analytics, AI/ML and more.

A few caveats before discussing results. Small objects do not saturate the network in the same way that large objects do. Performance for small object use cases is about the ability to conduct huge numbers of concurrent operations, it’s not about the total amount of data that gets pushed in a unit of time. Many object storage solutions aren’t capable of this degree of small object performance because they are limited by a reliance on a metadata database, and this is where MinIO’s ability to store objects in-line with metadata really sets it apart. Please see The Challenge in Big Data is Small Files for a detailed explanation of how MinIO is built to handle both small and large objects.

Since we usually present throughput results, we ran Performance Test, an automated test suite freely available to licensed MinIO customers, and results were simply marvelous – 105.39 GiB/s GET and 80.84 GiB/s PUT for the cluster. The results provided by Performance Test are best case. These results differ from those presented in the table below. For the results below, we used our more comprehensive test tool, WARP, which allows fine tuning of parameters like object size.

We ran the WARP S3 benchmark to measure READ/GET and WRITE/PUT performance of MinIO on the Supermicro Cloud DC cluster. WARP is an open source object storage performance benchmark tool developed and maintained by MinIO. We ran WARP with 4, 16, 64, 128 and 1024KiB object sizes to measure READ/GET and WRITE/PUT performance measured as throughput and objects/sec.

The results of our small files performance testing using WARP are summarized below:      

Dense Yet Powerful

Performance is a critical requirement for almost every object storage implementation. We have demonstrated multiple times over multiple years that MinIO is the fastest object store on the planet. MinIO takes complete advantage of the underlying hardware available to it, and excels at maxing out throughput with large objects and maxing out the number of objects/sec with small objects.  

MinIO dominates performance to be the best home for the most demanding use cases in streaming analytics, AI/ML and data lake analytics. When it comes to small file performance, no one even comes close. But we don’t stop there, in addition to better performance MinIO also provides a better developer experience with small files, enabling them to work with individual files in a ZIP file. This improves performance as well because it is faster to serve a single large ZIP than multiple small files, especially if they are transferred individually.

The Supermicro GrandTwin™ SuperServer is a solid, well-designed NVMe class hardware that we recommend for MinIO workloads. This is a dense, rack-optimized platform that shines in rack-dense environments, such as on-premise data centers, colocation facilities and the edge. Hardware like this is perfect for building your own small footprint private cloud or multi-tenant object storage as a service.  

If you want to discuss the results of this small file benchmark in detail or ask questions about benchmarking your environment, please join us on slack or email us at hello@min.io.