S3 - MinIO Blog

Model Checkpointing using Amazon’s S3 Connector for PyTorch and MinIO

Keith Pijanowski Keith Pijanowski on AI/ML | 17 January 2025

In November of 2023, Amazon announced the S3 Connector for PyTorch. The Amazon S3 Connector for PyTorch provides implementations of PyTorch's dataset primitives (Datasets and DataLoaders) that are purpose-built for S3 object storage. It supports map-style datasets for random data access patterns and iterable-style datasets for streaming sequential data access patterns. The S3 Connector for PyTorch also includes

The Innovations from AWS re:Invent

Keith Pijanowski Keith Pijanowski on AI/ML | 31 December 2024

Earlier this month, Amazon held their re:Invent conference in Las Vegas, Nevada, from December 1st to 5th - a 5-day event. If you have never been to a re:Invent conference, then the word that describes it best is “huge” - not just in terms of the number of attendees (60,000) but also the breadth of topics covered.

Iterable-Style Datasets using Amazon’s S3 Connector for PyTorch and MinIO

Keith Pijanowski Keith Pijanowski on AI/ML | 23 December 2024

In November of 2023 Amazon announced the S3 Connector for PyTorch. The Amazon S3 Connector for PyTorch provides implementations of PyTorch's dataset primitives (Datasets and DataLoaders) that are purpose-built for S3 object storage. It supports map-style datasets for random data access patterns and iterable-style datasets for streaming sequential data access patterns. In a previous post, I introduced the

Map-Style Datasets using Amazon’s S3 Connector for PyTorch and MinIO

Keith Pijanowski Keith Pijanowski on AI/ML | 31 October 2024

Before diving into Amazon’s S3 Connector for PyTorch, it is worthwhile to introduce the problem it is intended to solve. Many AI models need to be trained on data that cannot fit into memory. Furthermore, many really interesting models being built for computer vision and generative AI use data that cannot even fit on the disk drive that comes

Load Balancing with MinIO Firewall

AJ AJ on Elastic Load Balancers | 31 July 2024

The Load Balancer in MinIO Firewall solves the network bottleneck. In a cloud-native environment like Kubernetes, MinIO Firewall can be fairly easy to enable the Load Balancing without any modification to your application binary or container image.

Optimizing AI Data Processing with MinIO Weaviate and Langchain in Retrieval Augmented Generation (RAG) Pipelines

David Cannan David Cannan on AI/ML | 29 April 2024

Delve into AI’s next frontier with MinIO S3 Object-Store and SDK, enhancing a Weaviate Retreival Augmented Generation (RAG) Pipeline for robust data management. Discover how to elevate efficiency in AI systems using LangChain, unlocking new dimensions in scalable AI solutions.

Building and Deploying a MinIO-Powered LangChain Agent API with LangServe

David Cannan David Cannan on AI/ML | 9 April 2024

Explore the exciting possibilities of leveraging MinIO and LangChain to create a robust and efficient agent capable of handling complex data processing tasks.

Powering AI/ML Innovation: Building Feature Stores with MinIO’s High-Performance Object Storage

David Cannan David Cannan on AI/ML | 12 March 2024

MinIO’s high-performance object storage is key for AI innovation, offering scalability and integration for feature stores. Its capabilities enable seamless ML workflows, enhancing data management for AI development and deployment, impacting sectors like e-commerce and healthcare.

Developing Langchain Agents with the MinIO SDK for LLM Tool-Use

David Cannan David Cannan on AI/ML | 20 February 2024

Explore Langchain’s LLM Tool-Use and leverage Langgraph for monitoring MinIO’s S3 Object Store. This guide walks you through developing custom conversational AI agents and creating powerful OpenAI LLM chains for efficient data management and enhanced application functionality.

Prefix vs Folder

AJ AJ on Object Storage | 13 February 2024

How you ever wondered how object storage creates its folder structure mimicking a POSIX style hierarchy but something that is actually built for speed and efficiency? Today in this post you will find out what actually makes the internal structure you see visually in your MInIO buckets.

Backing Up Weaviate with MinIO S3 Buckets

David Cannan David Cannan on AI/ML | 6 February 2024

Explore integrating MinIO with Weaviate using Docker Compose for AI-enhanced data management. Learn to back up Weaviate to MinIO S3 buckets, ensuring data integrity and scalability with practical Docker and Python examples. Streamline your AI-driven search and analysis with this robust setup.

Innovating S3 Bucket Retrieval: Langchain Community S3 Loaders with OpenAI API

David Cannan David Cannan on AI/ML | 30 January 2024

Explore the synergy of MinIO, Langchain, and OpenAI in enhancing data storage and processing. This article illustrates MinIO’s integration for efficient document summarization using Langchain and OpenAI’s GPT, revolutionizing AI and ML data handling.

MinIO Gateway Migration

AJ AJ on Operator's Guide | 14 November 2022

This blog post will show you how you can migrate your existing MinIO Gateway setup to a MinIO Server deployment.

The Buyer’s Guide to Software Defined Object Storage

Jonathan Symonds Jonathan Symonds on Architect's Guide | 12 October 2022

Creating a framework for high-performance, cloud-native object storage is mission-critical in the modern enterprise. Take a look at The Buyer’s Guide to Software Defined #ObjectStorage to understand the key capabilities.

Using MinIO for AWS Local Zones

Jonathan Symonds Jonathan Symonds on S3 | 18 August 2022

As AWS Local Zones expand - object storage is needed. AWS recommends using MinIO.

Gateway Deprecation Implications for Azure Customers

Jonathan Symonds Jonathan Symonds on Azure | 17 August 2022

As we noted back in April, MinIO will be deprecating the gateway functionality in a few weeks time. As Harsha wrote at the time, the gateway had served its purpose and was no longer viable. While all gateway users will need to make some decisions, there are some Azure gateway users (for the cloud) and this post describes what those

How to Run Apache Druid and Apache Superset with MinIO

Matt Sarrel

Matt Sarrel @msarrel on Operator's Guide | 16 May 2022

How to Run Apache Druid and Apache Superset with MinIO

Learn how to build a cloud-native analytics and visualization stack backed by MinIO.

Deprecation of the MinIO gateway

Harshavardhana Harshavardhana on S3 | 24 February 2022

MinIO is deprecating the gateway and will be completely removed in six months. This should not come as a surprise, we began informing the community in 2020 and have steadily removed unpopular gateways. In the last ten months, MinIO has only made bug fixes. The community can continue to use older versions of MinIO past that date. We also encourage

How to Secure MinIO - Part 1

Andreas Auernhammer Andreas Auernhammer @_aead_ on Operator's Guide | 11 January 2022

Learn how to secure data in transit, data at-rest and establish role-based access control policies in the first of a series of blog posts about securing MinIO.

Mono Clouds vs Multi-Clouds & Hybrid Clouds

Jonathan Symonds Jonathan Symonds on Multicloud | 1 July 2021

What Amazon Web Services has done is truly remarkable. Over the better part of fifteen years they have built a business with an annual run rate in excess of $45B. Underpinning that remarkable growth is storage – specifically S3, launched back in 2006. S3 revolutionized the storage industry and the S3 API is now the standard in the cloud, replacing the