Before diving into Amazon’s S3 Connector for PyTorch, it is worthwhile to introduce the problem it is intended to solve. Many AI models need to be trained on data that cannot fit into memory. Furthermore, many really interesting models being built for computer vision and generative AI use data that cannot even fit on the disk drive that comes
Read more...
One of the biggest challenges facing organizations today for AI and data management is access to reliable infrastructure and compute resources. The Intel Tiber Developer Cloud is purpose-built for engineers who need an environment for proof-of-concepts, experimentation, model training, and service deployments. Unlike other clouds, which can be unapproachable and complex, the Intel Tiber Developer Cloud is simple and easy
Read more...
Parsec Labs is a company of engineers. Most have designed storage systems, been responsible for backups and replication, or worked in networking building switches. Founded in 2013, their Unified Data Mobility and Protection Appliance provides the most straightforward tools for migrating, replicating, and backing up data at scale. A Common Request As a one-time pre-sales engineer, Mark Clark, CEO of
Read more...
MinIO Enterprise Object Store is a foundational component for creating and executing complex data workflows. At the core of this event-driven functionality is MinIO bucket notifications using Kafka.
Read more...
Everybody claims to be a software company these days. From the nearly decade old pronouncement by Marc Andressen that “Software Is Eating the World” to the push from Wall Street to produce recurring software revenue; the pressure is on to claim - at least - that you are a software company. This is obviously problematic for appliance vendors. Try as
Read more...
Cloud-native AI/ML workloads push storage to the limit with many small files. MinIO Enterprise Object Store combines metadata and data to optimize small file operations.
Read more...
Unlock real-time analytics, scalability, and future-proof your data with a lakehouse. Hadoop can't keep up with AI, but a lakehouse blends the best of data lakes and warehouses. Get actionable tips to smoothly phase your migration with minimal disruption.
Read more...
We have said it before, but it bears repeating. The cloud is an operating model - not a physical location. That is why you will find MinIO everywhere on the public cloud, on the private cloud, at the edge. We don’t differentiate and because we are cloud native we are cloud (location) agnostic. The public cloud has mindshare and
Read more...
In this post we’ll show you how you visualize the cluster metrics in a web browser and also we’ll set up alerting so that when something like a drive needs to be replaced or drive runs out of space we can get alerted for it.
Read more...
To ensure AI success, start by hiring a data engineer, not an AI/ML expert. Learn from our experience and find out why a strong data foundation—focused on object storage, data lakehouses, and optimized pipelines—is critical for scalable, efficient AI/ML workloads.
Read more...
Tesla just open-sourced its Tesla Transport Protocol over Ethernet (TTPoE). Like Tesla’s earlier move to share EV patents, this opens the door to faster innovation through open-source contributions, proving once again that transparency drives progress.
Read more...
on
Getting started with MinIO is easy—with a few simple commands and a small 100 MB binary, you can have a functional development environment up and running in no time. However, to harness the full capability of MinIO at production scale, professionals are encouraged to learn more about MinIO’s extensive capabilities. We are introducing the MinIO Learning Subscription, featuring
Read more...
You’ve surely version controlled code in the past. But have you version controlled your data? Did you ever want to collaborate on large sets of data with various teams without committing a large chunk?
Read more...
Enhance your AI workflows by combining MinIO’s scalable Enterprise Object Storage with Polars, a lightning-fast DataFrame library. Learn how this powerful duo accelerates data pipelines, handles massive datasets, and offers powerful performance and scale.
Read more...
Dell generally focused on the filer game, but they dabble in object storage and have a very old offering, ECS. That makes sense, it was a step up from tape and wasn’t suited for dynamic workloads like HDFS modernization or database workloads. Needless to say, AI was out of the question. For a few years now, Dell has been
Read more...
Faced with skyrocketing compute costs, MinIO data scientist Archana Vaidyanathan leveraged the power of the data lakehouse, which allows for flexible compute choices without overhauling storage. MinIO Enterprise Object Storage enhances this model, delivering speed, scalability, and cost savings.
Read more...
Simplifying your data streaming architecture with WarpStream, a cloud-native, Kafka-compatible platform that cuts costs and complexity, recently acquired by Confluent. Paired with MinIO's high-performance object storage, it's a powerful alternative to Kafka for scalable, cost-effective streaming.
Read more...
Large numbers of small files present big challenges for application performance.
Read more...
Confluent's WarpStream acquisition highlights the future of data streaming built on object storage. WarpStream’s cloud-native design cuts costs by 85% over traditional Kafka. Believe the hype: object storage drives low-cost, scalable performance.
Read more...
Pairing the Iceberg table format with MinIO Enterprise Store creates a powerful, flexible and extensible lakehouse platform. The Iceberg Table Spec declares a table format that is designed to manage “a large, slow-changing collection” of files or objects stored in a distributed system.
Read more...
Erasure Code Calculator
Reference Hardware
