Object Storage is Primary Storage

The storage world has changed and applications are driving that change. Cloud native databases, analytics, AI/ML/DL and other applications are available to anyone with interest and are commonplace as enterprises seek to extract more value from their data.

These applications are forcing a migration to object storage as their primary storage. The reason is performance AT scale. Object storage is the only solution that delivers the combination of the two and as a result is the primary solution in the enterprise storage playbook.

Indeed, a recent IDC survey showed that 80% of respondents believe object storage can support their top IT initiatives.

Today’s applications are built using cloud architectures and paradigms and the S3 API is a key component of that. Software - databases, analytics apps, AI/ML apps - are written for object storage interfaces. It’s all about cloud-native microservices, containers and Kubernetes orchestration. There is no place for POSIX in this new world - despite the proliferation of “connectors.”

Make no mistake, performance is paramount. One can make a good argument that the main reason that object storage is now considered primary storage for cloud-native workloads is performance. MinIO’s combination of scalability and high-performance puts every workload, no matter how demanding, within reach. MinIO is capable of tremendous performance - a recent benchmark achieved 325 GiB/s on GETs and 165 GiB/s on PUTs with just 32 nodes of off-the-shelf NVMe SSDs. MinIO more than delivers the performance needed to power demanding workloads like Apache Spark, Starburst Presto/Trino, Clickhouse, and just about any other cloud-native framework you can think of.

The importance of using NVMe SSD and flash memory to propel object storage to higher speeds and lower latencies should not be underestimated. The NVMe protocol was built from the ground up transfer data between devices using the PCIe bus, where they can use up to 1 GB/s per lane with up to 16 lanes, providing the speed needed to support today’s multicore CPUs and high-speed RAM. In contrast to older technologies like SCSI, SATA and SAS (the latter only capable of supporting 256 commands on a single queue), NVMe is capable of supporting 64K commands per queue in up to 64K queues. Moreover, I/O commands and responses take place on the same processor core to take advantage of parallel processing on multi-core processors, enabling massive parallelization. NVMe is the perfect foundation for object storage to back demanding workloads across a fast 100 GbE datacenter network.  

Enterprises are adopting NVMe drives and flash at a tremendous pace to meet demand for applications that require massive amounts of data at very high speed.. The global NVMe market is estimated to be $44.6 billion in 2020 and projected to grow at a CAGR of almost 30% from 2020 to 2025. Enterprises can’t buy the hardware fast enough - the significant performance improvements of NVMe are driving a steep decline in the price/performance gap. Object storage running on NVMe flash and SSD servers and connected to applications over fast 100GbE is the number one choice for enterprise data centers today because it delivers low-latency access, very high IOPS and high throughput reads/writes, all while consuming less power.    

Object Storage for Flexibility, Performance and Scalability

Let’s take a look at three uses cases for primary storage that have been taken over by object storage:

Databases

The availability of high-performance and elastically scalable object storage has risen to the challenges presented by the explosive growth of both structured and unstructured data. Object storage must deliver exceptional throughput at an acceptable latency in order to provide backing storage to a database. The best database architectures are disaggregated, with developers heavily focused on the compute side of distributed low-latency query processing and letting object storage like MinIO handle the rest.

OS images, Application artifacts, Snapshots, Backups

Enterprises have wanted faster backup and restore operations for decades, but couldn’t achieve them because of the emphasis on the durability and low cost of backup infrastructure. High-performance object storage allows applications to take advantage of disaggregation to deliver superior performance, scalability and economics. Cloud-native applications read and write to object storage like MinIO using the S3 API over https, making it a convenient, consistent and portable solution for saving artifacts. When relying on MinIO as a ransomware protection mechanism, you gain the ability to conduct a very fast restore from an immutable snapshot, minimizing operational disruption.

AI/ML/DL and Stream/Log Analytics

We live in a data-driven and Kubernetes-dominated world. Cutting edge applications for real-time analytics and visualization rely on event streaming and require object storage as a sink. The AI (ML/DL) workloads that drive the most business value today rely on object storage for multiple pipeline stages and for model storage. HPC, formerly the realm of scale-up, has embraced the scale-out architecture of the cloud, depending on object storage for performance at scale.

It’s Prime Time!

As more cloud native applications are developed, object storage is increasingly used as primary storage because of its portability, performance and scalability. This is particularly true when leveraging NVMe SSD and flash as storage. In addition, the S3 API is the primary way for cloud-native applications to access data. MinIO gives enterprises the ability to run wherever object storage is needed and is already integrated with leading software - and the list grows every day.  

Take the first step into object storage and download MinIO today. Learn more about NVMe storage options on our reference hardware page. If you have any questions about getting the most out of MinIO in your environment, ping us on hello@min.io or join our Slack community.