Modern Data Lakes - MinIO Blog

From Data Swamps to Reliable Data Systems: How Iceberg Brought 40 Years of Database Wisdom to Data Lakes

Karthic Rao @hackintoshrao on Apache Iceberg | 25 July 2025

From Data Swamps to Reliable Data Systems: How Iceberg Brought 40 Years of Database Wisdom to Data Lakes

The data lake was once heralded as the future, an infinitely scalable reservoir for all our raw data, promising to transform it into actionable insights. This was a logical progression from databases and data warehouses, each step driven by the increasing demand for scalability. Yet, in embracing the data lake's scale and flexibility, we overlooked a critical difference.

Building Real-time Data Pipelines with MinIO's AIStor

Samson Koshy Samson Koshy on Data Engineering | 23 May 2025

Built a portable Java data pipeline using MinIO's AIStor and Kafka that scales from Mac to Kubernetes. Containerized stack (Kafka, AIStor, Prometheus, Grafana) processes millions of events, preserving raw data for analysis while delivering real-time dashboard summaries.

Managing AI workloads with Tagging and Policies

AJ AJ on Modern Data Lakes | 30 October 2024

Object tags give you greater power. You now have the ability to categorize by up to ten dimensions. If you want to add the diagram to a project, then all you have to do is tag it appropriately.

Hiring for AI Success: Why Your First Hire Should Be a Data Engineer

Brenna Buuck

Brenna Buuck on AI/ML | 4 October 2024

Hiring for AI Success: Why Your First Hire Should Be a Data Engineer

To ensure AI success, start by hiring a data engineer, not an AI/ML expert. Learn from our experience and find out why a strong data foundation—focused on object storage, data lakehouses, and optimized pipelines—is critical for scalable, efficient AI/ML workloads.

The Power of Flexibility in the Modern Data Stack: The Data Lakehouse Advantage

Brenna Buuck

Brenna Buuck on Modern Data Lakes | 24 September 2024

The Power of Flexibility in the Modern Data Stack: The Data Lakehouse Advantage

Faced with skyrocketing compute costs, MinIO data scientist Archana Vaidyanathan leveraged the power of the data lakehouse, which allows for flexible compute choices without overhauling storage. AIStor enhances this model, delivering speed, scalability, and cost savings.

The Challenge in Big Data is Small Files

Matt Sarrel

Matt Sarrel @msarrel on Architect's Guide | 20 September 2024

The Challenge in Big Data is Small Files

Large numbers of small files present big challenges for application performance.

ACID Transactions with Iceberg on AIStor

AJ AJ on Apache Iceberg | 18 September 2024

Pairing the Iceberg table format with AIStor creates a powerful, flexible and extensible lakehouse platform. The Iceberg Table Spec declares a table format that is designed to manage “a large, slow-changing collection” of files or objects stored in a distributed system.

Leading the Way: MinIO's Conditional Write Feature for Modern Data Workloads

Brenna Buuck

Brenna Buuck on Modern Data Lakes | 16 September 2024

Leading the Way: MinIO's Conditional Write Feature for Modern Data Workloads

MinIO introduced its conditional write feature long before AWS S3’s recent announcement. This powerful tool offers greater control in high-concurrency environments, ensuring data consistency and reliability, especially in AI and ML workflows.

Stop Giving Your Data to Vendors

Brenna Buuck

Brenna Buuck on Databricks | 11 September 2024

Databrick's CEO Ali Ghodsi Decouple storage and compute for more control, lower costs, and scalability. Modern datalakes, built on high-performance object storage like MinIO, empower you to handle AI/ML workloads with flexibility and performance—without relying on proprietary platforms.

Migrating from HDFS to AIStor

AJ AJ on Modern Data Lakes | 11 September 2024

Take advantage of cloud native, Kubernetes-oriented, microservices-based architectures with object storage.

The Bank of the East - Replacing Hadoop with MinIO and Dremio

Jonathan Symonds Jonathan Symonds on Case Study | 17 August 2024

Our client, a global financial institution headquarterd in Japan, recently completed an ambitious Hadoop replacement project with MinIO and Dremio. You can see them present it in this talk from Subsurface but we thought we would write it up as well. Like most banks, the firm had built out a large Hadoop footprint to power its analytics and risk management

PostgreSQL Meets Object Storage: Access External Data in MinIO

Brenna Buuck

Brenna Buuck on PostgreSQL | 15 August 2024

PostgreSQL Meets Object Storage: Access External Data in MinIO

The rise of lakehouse functionality is reshaping data management. ParadeDB's pg_lakehouse extension lets PostgreSQL integrate with object storage, enabling scalable, secure analytics. This makes the modernization of data infrastructure possible without extensive overhauls. Welcome to the future!

The Foundation of the Modern Datalake: How Object Storage Anchors Everything

Brenna Buuck

Brenna Buuck on Modern Data Lakes | 8 August 2024

The Foundation of the Modern Datalake: How Object Storage Anchors Everything

Amid the AI frenzy, one silent hero powers it all: modern object storage. It may not be glamorous, but it's the backbone of today's data lakes, enabling vast, efficient data management. Discover how AIStor elevates your data infrastructure.

The Catalog’s “IT” Moment and What it Means For Object Storage and AI

Brenna Buuck

Brenna Buuck on Modern Data Lakes | 25 July 2024

The Catalog’s “IT” Moment and What it Means For Object Storage and AI

Catalogs are revolutionizing modern datalakes, with industry giants like Databricks and Snowflake adopting Apache Iceberg’s catalog REST API. A commitment to open standards enhances performance, fosters innovation, and transforms data management for AI and ML.

The Architect's Guide to the New Private Cloud

Ugur Tigli Ugur Tigli on Modern Data Lakes | 22 July 2024

This post initially appeared on The New Stack. For a few years there, the term “private cloud” had a negative connotation. But as we know, technology is more of a wheel than an arrow, and right on cue, the private cloud is getting a ton of attention and it is all positive. The statistics are clear, Forrester’s 2023 Infrastructure

Enhancing Modern Datalakes with a Robust Semantic Layer

Brenna Buuck

Brenna Buuck on Modern Data Lakes | 17 July 2024

Enhancing Modern Datalakes with a Robust Semantic Layer

The semantic layer in modern datalakes provides context and structure to raw data, crucial for key data initiatives like AI model training, data management and data governance. A unified strategy and robust infrastructure are essential for effective implementation of the semantic layer.

Architecting a Modern Data Lake

Raghav Karnam

Raghav Karnam on Architect's Guide | 16 July 2024

The Modern Datalake is one-half data warehouse and one-half data lake and uses object storage for everything. The use of object storage to build a data warehouse is made possible by Open Table Formats OTFs) like Apache Iceberg, Apache Hudi, and Delta Lake, which are specifications that, once implemented, make it seamless for object storage to be used as the

Earn your RAG-ing rights with MinIO

Dileeshvar Radhakrishnan

Dileeshvar Radhakrishnan , AJ AJ on AI/ML | 26 June 2024

In this blog, we will demonstrate how to use MinIO to build a Retrieval Augmented Generation(RAG) based chat application using commodity hardware.

Latest Enhancements to Snowflake External Tables: What You Need to Know

Brenna Buuck

Brenna Buuck on Snowflake | 23 May 2024

Latest Enhancements to Snowflake External Tables: What You Need to Know

Snowflake's support for external tables has seen significant updates since our last blog post on how to extend your Snowflake implementation with MinIO. External tables allow users of Snowflake to treat data in object storage like MinIO as a read-only table in Snowflake without migration. Snowflake's ongoing enhancements to their external table functionality clearly demonstrate the

Deploy MinIO and Trino with Kubernetes

AJ AJ on AI/ML | 8 May 2024

In this tutorial, we'll deploy a cohesive system that allows distributed SQL querying across large datasets stored in Minio, with Trino leveraging metadata from Hive Metastore and table schemas from Redis.