Disaggregation, Analytical Engines and Starburst Trino
With today’s announcement of Starburst’s support for MinIO, it made sense to revisit the architectural trends that are becoming the standard for analytics workloads. Starburst provides a perfect example as we shall see shortly.
The architecture follows the model of disaggregating storage and compute. Modern, high speed networks have obsoleted the old approaches espoused by the many defunct HDFS vendors - that of bringing your data to your compute.
Starburst’s Trino (formerly PrestoSQL) is a SQL query engine - not a SQL database. Starburst has eschewed the storage component of the SQL database to focus on only one thing - ultra-fast SQL querying. Trino is just a query engine and does not store data. Instead, Trino interacts with various databases or directly on object storage. Trino parses and analyzes the SQL query you pass in, creates and optimizes a query execution plan that includes the data sources, and then schedules worker nodes that are able to intelligently query the underlying databases they connect to.
Trino is special because it optimizes itself for the data layer - be it databases with special indexes and specific formats or object stores. The goal in most optimizations for Trino is to push down the query to only get back the smallest amount of data needed to join with another dataset from another database or object store, do some further Trino specific processing, or simply return as the correct result set for the query.
In the case of Starburst, and in the case of many other database applications - performance must be available at every component - the compute/analytics layer, the network and, obviously, the data layer.
Performance
The storage performance requirements associated with analytic workloads have inverted in the past few years. What was once an IOPS problem became a throughput problem. That has led to the rise of object storage. Object storage is throughput driven, not IOPS or latency driven.
In the case of Starburst - they demand performance for their data store - which is increasingly object storage oriented. Starburst queries the data directly where it lives. This can be "dimensional" type data in a relational database (MySQL) and/or data that is joined with raw data in S3. Both the databases and the raw data can and do leverage object storage. This saves them time and money because they do not have to move data into other products such as Amazon Redshift.
In the case of MinIO, we are not just the world’s fastest object store in terms of throughput but MinIO also excels in small objects performance where the table segments range from 256K to 2MB.
The reason for this is important. MinIO does not use a metadata database because of its deterministic hashing to look up objects. Other object storage implementations quickly get overwhelmed when you store these table segments as small objects. This make an already fast Trino query even faster.
Scalability
Most file and block systems were not designed for scale. As a result, when you push past 100TB tradeoffs start to be made. When you get to 1PB - you are in rarefied air for these systems.
Object storage on the other hand, just starts hitting its sweet spot at 1PB. The sky is the limit from there. This makes object storage the ideal complement for analytical engines like Starburst with ambitions to deliver against giant application workloads that cover large components of an organization's data.
The model is simple given the throughput capabilities of fast object storage. A small portion of data is kept “in memory” for ultra-fast processing while the vast majority of it sits in a really, really warm tier - available using the standard S3 calls that have come to define the modern application ecosystem.
Proofpoint
A great example can be found in our earlier work with Starburst. Our benchmarks remain the standard for Presto performance and would easily be even faster today.
Conclusion
The S3 API is the de facto standard for storage these days. POSIX is on the decline - and there is no reason to believe it will recover.
As a result, leaders in the analytics space like Starburst have made supporting MinIO a priority - especially given the multi cloud capabilities we offer. MinIO is unique in its ability to run in AWS, GCP, Azure, OpenShift, Tanzu, baremetal, the edge. The consistency of the interface and the depth of the performance make it a key part of any large scale architecture.
The result is an effective separation of storage and compute with best of breed for both elements. We encourage you to try it out for yourself. You can download MinIO here and pick up docs info from Starburst.