In November of 2023, Amazon announced the S3 Connector for PyTorch. The Amazon S3 Connector for PyTorch provides implementations of PyTorch's dataset primitives (Datasets and DataLoaders) that are purpose-built for S3 object storage. It supports map-style datasets for random data access patterns and iterable-style datasets for streaming sequential data access patterns.
The S3 Connector for PyTorch also includes
Read more
Earlier this month, Amazon held their re:Invent conference in Las Vegas, Nevada, from December 1st to 5th - a 5-day event. If you have never been to a re:Invent conference, then the word that describes it best is “huge” - not just in terms of the number of attendees (60,000) but also the breadth of topics covered.
Read more
In November of 2023 Amazon announced the S3 Connector for PyTorch. The Amazon S3 Connector for PyTorch provides implementations of PyTorch's dataset primitives (Datasets and DataLoaders) that are purpose-built for S3 object storage. It supports map-style datasets for random data access patterns and iterable-style datasets for streaming sequential data access patterns.
In a previous post, I introduced the
Read more
Before diving into Amazon’s S3 Connector for PyTorch, it is worthwhile to introduce the problem it is intended to solve. Many AI models need to be trained on data that cannot fit into memory. Furthermore, many really interesting models being built for computer vision and generative AI use data that cannot even fit on the disk drive that comes
Read more
The Load Balancer in MinIO Firewall solves the network bottleneck. In a cloud-native environment like Kubernetes, MinIO Firewall can be fairly easy to enable the Load Balancing without any modification to your application binary or container image.
Read more
Delve into AI’s next frontier with MinIO S3 Object-Store and SDK, enhancing a Weaviate Retreival Augmented Generation (RAG) Pipeline for robust data management. Discover how to elevate efficiency in AI systems using LangChain, unlocking new dimensions in scalable AI solutions.
Read more
Explore the exciting possibilities of leveraging MinIO and LangChain to create a robust and efficient agent capable of handling complex data processing tasks.
Read more
MinIO’s high-performance object storage is key for AI innovation, offering scalability and integration for feature stores. Its capabilities enable seamless ML workflows, enhancing data management for AI development and deployment, impacting sectors like e-commerce and healthcare.
Read more
Explore Langchain’s LLM Tool-Use and leverage Langgraph for monitoring MinIO’s S3 Object Store. This guide walks you through developing custom conversational AI agents and creating powerful OpenAI LLM chains for efficient data management and enhanced application functionality.
Read more
How you ever wondered how object storage creates its folder structure mimicking a POSIX style hierarchy but something that is actually built for speed and efficiency? Today in this post you will find out what actually makes the internal structure you see visually in your MInIO buckets.
Read more
Explore integrating MinIO with Weaviate using Docker Compose for AI-enhanced data management. Learn to back up Weaviate to MinIO S3 buckets, ensuring data integrity and scalability with practical Docker and Python examples. Streamline your AI-driven search and analysis with this robust setup.
Read more
Explore the synergy of MinIO, Langchain, and OpenAI in enhancing data storage and processing. This article illustrates MinIO’s integration for efficient document summarization using Langchain and OpenAI’s GPT, revolutionizing AI and ML data handling.
Read more
This blog post will show you how you can migrate your existing MinIO Gateway setup to a MinIO Server deployment.
Read more
Creating a framework for high-performance, cloud-native object storage is mission-critical in the modern enterprise. Take a look at The Buyer’s Guide to Software Defined #ObjectStorage to understand the key capabilities.
Read more
As AWS Local Zones expand - object storage is needed. AWS recommends using MinIO.
Read more
As we noted back in April, MinIO will be deprecating the gateway functionality
in a few weeks time. As Harsha wrote at the time, the gateway had served its
purpose [https://blog.min.io/deprecation-of-the-minio-gateway/] and was no
longer viable.
While all gateway users will need to make some decisions, there are some Azure
gateway users (for the cloud) and
Read more
Learn how to build a cloud-native analytics and visualization stack backed by MinIO.
Read more
MinIO is deprecating the gateway and will be completely removed in six months.
This should not come as a surprise, we began informing the community in 2020 and
have steadily removed unpopular gateways. In the last ten months, MinIO has only
made bug fixes.
The community can continue to use older versions of MinIO past that date. We
also encourage
Read more
Learn how to secure data in transit, data at-rest and establish role-based access control policies in the first of a series of blog posts about securing MinIO.
Read more
What Amazon Web Services has done is truly remarkable. Over the better part of
fifteen years they have built a business with an annual run rate in excess of
$45B. Underpinning that remarkable growth is storage – specifically S3, launched
back in 2006. S3 revolutionized the storage industry and the S3 API is now the
standard in the cloud, replacing the
Read more
Replication
Versioning
Observability
Encryption
Key Management Server
S3 Compatibility
Object Immutability
Catalog
Object Prompt
Identity + Access Mgt
Firewall
AIHub
Lifecycle
Cache
Events and Lambdas
Console
Erasure Code Calculator
Reference Hardware
Integrations with MinIO
Download MinIO Object Store
Events
