AI/ML Best Practices During a Gold Rush

AI/ML Best Practices During a Gold Rush

Introduction The California Gold Rush started in 1848 and lasted until 1855. It is estimated that approximately 300,000 people migrated to California from other parts of the United States and abroad. Economic estimates suggest that, on average, only half made a modest profit. The other half either lost money or broke even. Very few gold seekers made a significant

Read more...

Parallel ML Experimentation leveraging MinIO & lakeFS

Parallel ML Experimentation leveraging MinIO & lakeFS

Introduction This post was written in collaboration with Iddo Avneri from lakeFS. Managing the growing complexity of ML models and the ever-increasing volume of data has become a daunting challenge for ML practitioners. Efficient data management and data version control are now critical aspects of successful ML workflows. In this blog post, we delve into the power of parallel ML

Read more...

Setting up a Development Machine with MLFlow and MinIO

Setting up a Development Machine with MLFlow and MinIO

About MLflow MLflow is an open-source platform designed to manage the complete machine learning lifecycle. Databricks created it as an internal project to address challenges faced in their own machine learning development and deployment processes. MLflow was later released as an open-source project in June 2018. As a tool for managing the complete lifecycle, MLflow contains the following components. * MLflow

Read more...

Enhance Large Language Models Leveraging RAG and MinIO on cnvrg.io

Enhance Large Language Models Leveraging RAG and MinIO on cnvrg.io

This post was written in collaboration with Harinder Mashiana from cnvrg.io. Large language models (LLMs) have revolutionized the world of technology, offering powerful capabilities for text analysis, language translation, and chatbot interactions. The revolution will heavily impact businesses, according to OpenAI, approximately 80% of the U.S. workforce could have at least 10% of their work tasks affected by

Read more...

Object Management for AI/ML

Object Management for AI/ML

Introduction In a few previous posts on AI/ML, I mentioned that one of the benefits of MinIO is that you have tools for Versioning, Lifecycle Management, Object Locking, Object Retention and Legal Holds. These capabilities have a variety of uses. You may need a simple way to keep track of training experiments. You could also use these features to

Read more...

The Architect’s Guide to Storage for AI

The Architect’s Guide to Storage for AI

This post first appeared in The New Stack. Developers gravitate to technologies that are software defined, open source, cloud native and simple. That essentially defines object storage. Introduction Choosing the best storage for all phases of a machine learning (ML) project is critical. Research engineers need to create multiple versions of datasets and experiment with different model architectures. When a

Read more...

Machine Learning Using H20, R and MinIO

Machine Learning Using H20, R and MinIO

I’ve been working with neural networks and machine learning since the late ‘80. Yes, I’m that old. The first product I bought was California Scientific Software BrainMaker Professional. I loved that product because it got me started with neural networks. I still have it: It was a 3 layer neural network product that came with source code in

Read more...

The Architect’s Guide to Using AI/ML with Object Storage

The Architect’s Guide to Using AI/ML with Object Storage

This post first appeared in The New Stack. With the constant evolution of the enterprise, machine learning and artificial intelligence have become board-level initiatives. Marketing claims aside, capabilities that seemed almost mythical a few years ago are now taken for granted as AI/ML becomes baked into every software stack and architecture. This is becoming known as AI-first architecture. In

Read more...

How to Install and Configure Kubeflow with MinIO Operator

How to Install and Configure Kubeflow with MinIO Operator

Kubeflow is a modern solution to design, build and orchestrate Machine Learning pipelines using the latest and most popular frameworks. Out of the box, Kubeflow ships with MinIO inside to store all of its pipelines, artifacts and logs, however that MinIO is limited to a single PVC and thus cannot benefit from all the features a distributed MinIO brings to

Read more...

Hyper-Scale Machine Learning with MinIO and TensorFlow

Hyper-Scale Machine Learning with MinIO and TensorFlow

We are living in a transformative era defined by information and AI. Massive amounts of data are generated and collected every day to feed these voracious, state-of-the-art, AI/ML algorithms. The more data, the better the outcomes. One of the frameworks that has emerged as the lead industry standards is Google's TensorFlow. Highly versatile, one can get started

Read more...