A Closer Look: MinIO Observability
Observability is all about gathering information (traces, logs, metrics) with the goal of improving performance, reliability, and availability.
Read moreMinIO Blog
Bringing ARM into the AI Data Infrastructure Fold at MinIO Using SVE
One of the reasons that MinIO is so performant is that we do the granular work that others will not or cannot. From SIMD acceleration to the AVX-512 optimizations we have done the hard stuff. Recent developments for the ARM CPU architecture, in particular Scalable Vector Extensions (SVE), presented us with the opportunity to deliver significant performance and efficiency gains
Read moreThe Architect's Guide to the New Private Cloud
This post initially appeared on The New Stack. For a few years there, the term “private cloud” had a negative connotation. But as we know, technology is more of a wheel than an arrow, and right on cue, the private cloud is getting a ton of attention and it is all positive. The statistics are clear, Forrester’s 2023 Infrastructure
Read moreEnhancing Modern Datalakes with a Robust Semantic Layer
The semantic layer in modern datalakes provides context and structure to raw data, crucial for key data initiatives like AI model training, data management and data governance. A unified strategy and robust infrastructure are essential for effective implementation of the semantic layer.
Read moreThe App Store of OpenShift: MinIO in OperatorHub
Simply put, OperatorHub to OpenShift is what App Store is to Apple. With a web console interface, an Operator can be pulled from its off-cluster source, installed and subscribed on the cluster, and made ready for engineering teams to self-service manage the product across deployment environments.
Read moreArchitecting a Modern Data Lake
The Modern Datalake is one-half data warehouse and one-half data lake and uses object storage for everything. The use of object storage to build a data warehouse is made possible by Open Table Formats OTFs) like Apache Iceberg, Apache Hudi, and Delta Lake, which are specifications that, once implemented, make it seamless for object storage to be used as the
Read moreData-Centric AI with Snorkel and MinIO
With all the talk in the industry today regarding large language models with their encoders, decoders, multi-headed attention layers, and billions (soon trillions) of parameters, it is tempting to believe that good AI is the result of model design only. Unfortunately, this is not the case. Good AI requires more than a well-designed model. It also requires properly constructed training
Read moreMinIO hits it out of the Boundary
Boundary helps record SSH sessions to meet compliance and improve security requirements. These sessions are then stored on MinIO for fast retrieval for auditing purposes in case of a data breach incident.
Read moreThe Significance of Databricks' Acquisition of Tabular: A Triumph for Open Frameworks in Data
Databricks' acquisition of Tabular, founded by the creators of Apache Iceberg, underscores the importance of open frameworks in modern data lake design. Open frameworks ensure interoperability, flexibility, and simplicity, benefiting those leveraging data for AI.
Read moreMigrate to AI-Ready infrastructure: Hitachi Content Platform to MinIO
Migrate from Hitachi Content Platform (HCP) to MinIO using the HCP-to-MinIO tool. Migration is a no-brainer given how MinIO offers modern, scalable, high-performance storage optimized for AI.
Read moreEarn your RAG-ing rights with MinIO
In this blog, we will demonstrate how to use MinIO to build a Retrieval Augmented Generation(RAG) based chat application using commodity hardware.
Read moreThe Real Reasons Why AI is Built on Object Storage
tl;dr: In this post, we will explore four technical reasons why AI workloads rely on high performance object store. 1. No Limits on Unstructured Data In the current paradigm of machine learning, performance and ability scales with compute, which is really a proxy for dataset size and model size (Scaling Laws for Neural Language Models, Kaplan et. al.). Over
Read moreThe Architect’s Guide to the GenAI Tech Stack - Ten Tools
This post first appeared on The New Stack on June 3rd, 2024. I previously wrote about the modern data lake reference architecture, addressing the challenges in every enterprise — more data, aging Hadoop tooling (specifically HDFS) and greater demands for RESTful APIs (S3) and performance — but I want to fill in some gaps. The modern data lake, sometimes referred to as
Read moreWARP speed your AI data storage Infrastructure
Do you know the secret to some of the best AI models out there? It's the amount of data they had access to on which they could be trained on. For AI/ML models Fast accessible Data is King. Let me emphasize, it's not just Data, but fast accessible Data.
Read moreDell ECS Data Movement to MinIO
Dell ECS's “Data Movement”, also called copy-to-cloud is a feature introduced in ECS 3.8.0.1 that allows you to copy objects from Dell ECS to MinIO which is rather popular with customers and prospects who are modernizing their storage stack to support their AI data infrastructure requirements.
Read moreBoosting Snowflake with External Tables
Unlock Snowflake's potential by integrating external tables with MinIO. Seamlessly query external data without migration, boost analytics, save costs, and simplify access. This setup provides real-time insights and maximizes your infrastructure investment for both MinIO and Snowflake.
Read moreThe Economics of Public Cloud Repatriation
What has become clear over the past couple of years is that the public cloud, for all of its benefits, doesn't deliver cost savings at scale. It delivers productivity gains, to a point, but it will not reduce your costs. There is goodness in the public cloud as it offers an incredibly powerful value proposition—infrastructure available immediately,
Read moreIntegrate MinIO with Keycloak OIDC
In this blog post, we’ll show you how to set up MinIO to work with Keycloak. But broadly it should also give you an idea of how OIDC is configured with MinIO so you can use it with anything other than Keycloak, here we just use it as an example.
Read moreThe Future of Hybrid Cloud Pipelines: Integrating MinIO, Tailscale, and GitHub Actions
Streamline your data processing capabilities, ensuring high-quality data management and secure operations. This integration not only enhances workflow automation but also leverages the advanced functionalities of MinIO and Tailscale, providing a powerful solution for modern data processing needs.
Read moreLatest Enhancements to Snowflake External Tables: What You Need to Know
Snowflake's support for external tables has seen significant updates since our last blog post on how to extend your Snowflake implementation with MinIO. External tables allow users of Snowflake to treat data in object storage like MinIO as a read-only table in Snowflake without migration. Snowflake's ongoing enhancements to their external table functionality clearly demonstrate the
Read more
