Towards Exascale AI Data Infrastructure

It's been just over a week for me here at MinIO. The big takeaway from immersing myself in whiteboarding sessions, architecture reviews and customer calls is that the simplicity of the product is both its distinguishing feature and one of its most defining value drivers. This is particularly true at scale. The explosive growth in computing power due to AI advancements has significantly influenced the data landscape, ushering us into the era of exascale data infrastructure.

There are some amazing resources/recipes for success out there for founders and product leaders - but they all can be distilled into something that remains oddly elusive for most startups - “solve an urgent, pervasive problem that customers are willing to pay for.”

In the case of MinIO, that problem is about solving the challenges of large scale data infrastructure for dynamic workloads. That is AI/ML, that is SEIM/threat hunting data lakes, that is HDFS replacements, that is database persistent storage, that is advanced analytics and about a dozen more things. On the AI front, every enterprise company, irrespective of the size of the company, is trying to re-invent themselves using AI. The key metric is time to value from the AI initiative. To achieve this, three things need to come together. 

• The right data, in the right time window.
• The right infrastructure to serve, store and stream that data in a performant manner. 
• The right AI application(s) leveraging that data to achieve the desired business outcomes.

Let’s start with the right data in the right time window. The right data has X attributes. The right time window has Y attributes. Collectively they are the foundation of your data initiative - AI or otherwise and they compound in importance in the world of AI because, if your data is wrong you will perpetuate wrongness, you will teach wrongness and you will need to start over. 

Next up is the right infrastructure. My colleague Keith Pijanowski has written a stellar piece on the AI Data Lake that is worth your time. The AI ecosystem revolves around object storage. OpenAI is trained on an object store. Mistral is trained on an object store. Anthropic’s Claude is trained on an object store. 

The Cloud model paved the way for breakthroughs in AI. This is alarming for Legacy data storage companies that offer NAS (Network Attached Storage). They are at risk of going the way of tape in the age of AI. As a result, they are trying to stay relevant by providing an object store interface to the existing data via File Object Duality. It is the wrong approach - architecturally because this approach results in having an object store gateway interface to files as opposed to providing a true native object store. Such a solution unsuccessfully tries to retrofit AI workload needs to NAS workload capabilities but the only real goal is to make sure data does not leave their appliance. This operating model does not satisfy the required speed, simplicity, cost and scale that AI workloads demand irrespective of the hardware these storage appliance vendors offer.Modern AI applications are built using cloud native technologies. Most of these AI application workloads by default require a scale out distributed, highly available object store supporting un-structured/semi-structured data from a data persistence stand point. MinIO was built from the ground up to exactly cater to these cloud native AI application needs. Whether its bare-metal or containers, AIStor is truly a software defined cloud native Object storage powering for these applications.    

The proof is in the numbers. MinIO’s Docker Pull #s are up 84% YOY and just set a record at 1.6M per day. Let that sink in for a moment. 1.6M per day. That is likely more than all of our non-public cloud competitors combined in a quarter! Many of these Docker Pulls are from enterprises that are trying to leverage the cloud-native, “just works” simplicity of MinIO.

To achieve that level of success, it requires a better mousetrap. Keep in mind, every single company adopting MinIO already has some type of storage - even if it is legacy file and block. 

There is another dynamic at play here- and that is cloud repatriation. Enterprise customers are keen to avoid the mistakes they made in the rush to the public cloud - with lockin and uncontrolled bills. The data required to train your AI is bigger and the compute along with network egress cost becomes more expensive when working with these large data sets. Controlling these cloud costs while offering elasticity for the business is what the requirement is - and this is achievable today by moving to a colo provider like Equinix or back onto a private data center where customers are looking to save costs - as much as 50% or more while leveraging MinIO for cloud native Object store irrespective of where they move their data from public clouds. 

In summary, we will continue to grow and become a primary data infrastructure for AI workloads helping customers to innovate with operational agility, security and delivering performance required for the new age AI workloads at scale. If you have any questions, please reach out to us at hello@min.io or hit us on Ask the Expert button.