Product
AiStor Logo
Overview
What is AIStor?
Features
Replication Replication Versioning Versioning Observability Observability Encryption Encryption Key Management Server Key Management Server S3 Compatibility S3 Compatibility Object Immutability Object Immutability Catalog Catalog Object Prompt Object Prompt xxx Identity + Access Mgt Firewall Firewall AIHub AIHub Lifecycle Lifecycle Cache Cache Events and Lambdas Events and Lambdas Console Console
Tools
Erasure Code Calculator Erasure Code Calculator Determine your raw and usable capacity across a range of erasure coding settings. Reference Hardware Reference Hardware MinIO’s recommended Configuration and reference hardware for building large scale data infrastructure.
Solutions
AI Storage Learn how MinIO is leading the AI storage market from its exclusive features to performance at scale. HDFS Migration Modernize and simplify your big data storage infrastructure with cloud native storage with AIStor. Snowflake Learn how to leverage Snowflake external tables to query data without having to move it.
Integrations Integrations with MinIO Browse our expansive list of
integrations with links to the relevant
documentation.
Veeam Learn how MinIO and Veeam have partnered deliver superior RTO and RPO. Modern Datalakes Learn how modern, multi-engine data lakeshouses depend on MinIO's AIStor. Equinix Repatriate your data onto the cloud you control with MinIO and Equinix. SQL Server Learn how to leverage SQL Server 2022 with MinIO to run queries on your data without having to move it. Commvault Learn how Commvault and MinIO are partnered to deliver performance at scale for mission critical backup and restore workloads. Hybrid Cloud Learn how enterprises use MinIO to build AI data infrastructure that runs on any cloud - public, private or colo. Splunk Find out how MinIO is delivering performance at scale for Splunk SmartStores. VMware Discover how MinIO integrates with VMware across the portfolio from the Persistent Data platform to TKG and how we support their Kubernetes ambitions.
Open Source
Github Community Explore, experiment, ask questions and contribute. Docs Development guides. Slack Channel Open Forum for discussing topics related to MinIO.
Download MinIO Object Store Download MinIO Object Store Get MinIO's open source server, client, and SDK.
Resources
Resources Home Browse our library of white papers, solution briefs, benchmarks and videos. Blog Product updates, company news, and educational content. Training Learn the MinIO Way and Become a Data Infrastructure Expert.
Events Events Join us to take advantage to meet with MinIO partners, employees, and subject-matter experts.
Partner Pricing
Login Download

AI/ML workflows with AIStor and Metaflow

AJ AJ on AI/ML
AI/ML workflows with AIStor and Metaflow

AIStor follows the ethos of starting small, even on your laptop and scaling up to a full blown production cluster. The reason for this is so that we want to make the initial entry point to using the application as painfree and simple as possible. This has the ability for the app to quickly gain traction and be used more often to test the use cases.

We’re happy to see other apps in the ecosystem follow this same model. In this case it's Metaflow. Metaflow is a human-friendly Python library that makes it straightforward to develop, deploy, and operate various kinds of data-intensive applications, in particular those involving data science, ML, and AI. Metaflow was originally developed at Netflix to boost the productivity of data scientists who work on a wide variety of projects, from classical statistics to state-of-the-art deep learning.

AIStor and Metaflow can be deployed anywhere on any type of infrastructure. In fact more often than not we have folks deploying AIStor and metaflow alongside each other as part of the deployment pipeline.

Integrate with AIStor

Integration with AIStor is pretty simple with Metaflow.

Download and launch MinIO AIStor on your Kubernetes cluster, we’ll launch metaflow in the same cluster as well.

Follow the instructions below to set up a bucket in AIStor.

<div>

  <script async src="https://js.storylane.io/js/v2/storylane.js"></script>

  <div class="sl-embed" style="position:relative;padding-bottom:calc(79.17% + 25px);width:100%;height:0;transform:scale(1)">

<iframe loading="lazy" class="sl-demo" src="https://app.storylane.io/demo/cesgrcyf9wnq?embed=inline" name="sl-embed" allow="fullscreen" allowfullscreen style="position:absolute;top:0;left:0;width:100%!important;height:100%!important;border:1px solid rgba(63,95,172,0.35);box-shadow: 0px 0px 18px rgba(26, 19, 72, 0.15);border-radius:10px;box-sizing:border-box;"></iframe>

  </div>

</div>

Next lets setup Metaflow, there are a few environment variables we need to setup to connect to AIStor

  • `METAFLOW_DEFAULT_DATASTORE` - Set this to s3
  • METAFLOW_DATASTORE_SYSROOT_S3 - set this to s3://<bucket_name>
  • METAFLOW_S3_ENDPOINT_URL - set this to AIStor endpoint http://<minio>:9000/

Once these variables are set, launch Metaflow.

It's as simple as that.

Grow based on need 

Metaflow, just like AIStor, is designed to grow as you need. Most data scientists and AI/ML engineers want to start off with test data on their local environment, without the complexities of the infrastructure, just to make sure the focus is on the product and not on scaling. But once the product is proven then it comes time to scale and the nuances that come with it.

If you have any questions AIStor be sure to reach out to us at hello@min.io.