Introduction This post was written in collaboration with Iddo Avneri from lakeFS. Managing the growing complexity of ML models and the ever-increasing volume of data has become a daunting challenge for ML practitioners. Efficient data management and data version control are now critical aspects of successful ML workflows. In this blog post, we delve into the power of parallel ML
Read more
When we announced the availability of MinIO on Red Hat OpenShift, we didn’t anticipate that demand would be so great that we would someday write a series of blog posts about this powerful combination. This combination is being rapidly adopted due to the ubiquitous nature of on-prem cloud and the need of large organizations wanting to bring their data
Read more
About MLflow MLflow is an open-source platform designed to manage the complete machine learning lifecycle. Databricks created it as an internal project to address challenges faced in their own machine learning development and deployment processes. MLflow was later released as an open-source project in June 2018. As a tool for managing the complete lifecycle, MLflow contains the following components. * MLflow
Read more
MinIO Jumbo improved backup performance by 15x using parallel uploads.
Read more
This post was a collaboration between Kevin Lambrecht of UCE Systems and Raghav Karnam The cloud operating model and specifically Kubernetes have become the standard for large scale infrastructure today. More importantly, they are evolving at an exceptional pace with material impacts to data science, data analytics and AI/ML. This transition has a significant impact on the Hadoop ecosystem.
Read more
MinIO has developed into a core building block for the media and entertainment industry. With a customer roster that includes the leading cable company, the biggest streaming company and dozens of companies up and down the stack we have added a number of different features in recent quarters. One of those is called the fan out feature and it is
Read more
Making the serving of your AI models more lightweight by leveraging the simplicity of MinIO’s object store. tl;dr MinIO object storage can be used as a ‘single source of truth’ for your machine learning models and, in turn, make serving with PyTorch Serve more efficient when managing changes to Large Language Models (LLMs). As always, sample code is
Read more
MinIO is built with speed and resiliency at the forefront, regardless of the type of environment you choose to run it on. Whether it's multi cloud, bare metal, cloud instances or even on-premise, MinIO is designed to run on AWS, GCP, Azure, colocated bare metal servers and Kubernetes distributions such as Red Hat OpenShift. MinIO runs just as
Read more
Learn how to troubleshoot object storage performance with low level system component testing.
Read more
The world of backup has entered a brave new world where traditional solutions still have utility but where the scale, speed of change and application landscape require different…radically different…approaches. This post seeks to lay out the challenges of this new world, where the line of demarcation exists and how to think about architecting a data protection framework that
Read more
This post was written in collaboration with Harinder Mashiana from cnvrg.io. Large language models (LLMs) have revolutionized the world of technology, offering powerful capabilities for text analysis, language translation, and chatbot interactions. The revolution will heavily impact businesses, according to OpenAI, approximately 80% of the U.S. workforce could have at least 10% of their work tasks affected by
Read more
Introduction In a few previous posts on AI/ML, I mentioned that one of the benefits of MinIO is that you have tools for Versioning, Lifecycle Management, Object Locking, Object Retention and Legal Holds. These capabilities have a variety of uses. You may need a simple way to keep track of training experiments. You could also use these features to
Read more
When purchasing storage, the emphasis is usually on media, but it may be even more important to consider access methods too. You will need to take storage protocols into account when designing and procuring infrastructure, especially when you leave legacy storage behind in order to migrate to cloud-native object storage. However, object storage relies on the S3 API for communications,
Read more
Introduction In a previous post, I covered Building an ML Data Pipeline with MinIO and Kubeflow v2.0. The data pipeline I created downloaded US Census data to a dedicated instance of MinIO. This is different from the MinIO instance Kubeflow Pipelines (KFP) uses internally. We could have tried to use KFP’s instance of MinIO - however, this is
Read more
Our latest YouTube training series is all about the MinIO Operator, which brings native support for deploying and managing MinIO deployments (“MinIO Tenants”) on a Kubernetes cluster. MinIO’s Mike Johnson (aka MJ) brings us through the 10-part video series to set the foundation of understanding Kubernetes before focusing on installing and configuring the MinIO Operator for Kubernetes, which will
Read more
In this post we look at how search, and specifically OpenSearch can help us identify patterns or see trends in our ever growing data.
Read more
This post first appeared in The New Stack. Developers gravitate to technologies that are software defined, open source, cloud native and simple. That essentially defines object storage. Introduction Choosing the best storage for all phases of a machine learning (ML) project is critical. Research engineers need to create multiple versions of datasets and experiment with different model architectures. When a
Read more
Most developers, engineers, architects and DevOps folks know MinIO. Not all know that the only thing we do is software-defined object storage. We don’t do file or block. We don’t offer a service, it is self-hosted. Our focus is singular. The result is that our object store is objectively, based on adoption, awards and customer feedback the best
Read more
Apache Kafka and Apache Spark are two leading technologies used to build the streaming data pipelines that feed data lakes and lake houses. At a really high level, Kafka streams messages to Spark where they are transformed into a format that can be read in by applications and saved to storage.
Read more
Build data pipelines with S3 to MinIO and MinIO to MinIO batch replication.
Read more
