AI/ML Reproducibility with lakeFS and MinIO

AI/ML Reproducibility with lakeFS and MinIO

This post was written in collaboration with Amit Kesarwani from lakeFS. The reality of running multiple machine learning experiments is that managing them can become unpredictable and complicated - especially in a team environment. What often happens is that during the research process, teams constantly change configuration and data between experiments. For example, try several training sets and several hyperparameter

Read more...

Parallel ML Experimentation leveraging MinIO & lakeFS

Parallel ML Experimentation leveraging MinIO & lakeFS

Introduction This post was written in collaboration with Iddo Avneri from lakeFS. Managing the growing complexity of ML models and the ever-increasing volume of data has become a daunting challenge for ML practitioners. Efficient data management and data version control are now critical aspects of successful ML workflows. In this blog post, we delve into the power of parallel ML

Read more...

Enhance Large Language Models Leveraging RAG and MinIO on cnvrg.io

Enhance Large Language Models Leveraging RAG and MinIO on cnvrg.io

This post was written in collaboration with Harinder Mashiana from cnvrg.io. Large language models (LLMs) have revolutionized the world of technology, offering powerful capabilities for text analysis, language translation, and chatbot interactions. The revolution will heavily impact businesses, according to OpenAI, approximately 80% of the U.S. workforce could have at least 10% of their work tasks affected by

Read more...

Security Advisory

Synopsis: A denial-of-service (DoS) possibility against the MinIO server was discovered and has been fixed in RELEASE.2019-08-07T01-59-21Z. Severity: Medium Who is affected: All users of the MinIO server version RELEASE.2019–03–06T22–47–10Z or newer are affected. Further, all users of the MinIO gateway version RELEASE.2019–03–06T22–47–10Z or newer using STS are affected

Read more...

Security Advisory

Synopsis: Unauthorized bucket access possibilities against the IAM implementation of MinIO server was discovered and has been fixed in RELEASE.2019–06–15T23–07–18Z. Severity: High Who is affected: All users of IAM feature of MinIO server and gateway are affected. It is highly recommended to upgrade. Recommended Action for Users: All users are advised to upgrade their Minio

Read more...

Security Advisory

Synopsis: Two different privilege escalation possibilities against the IAM implementation of MinIO server were discovered and has been fixed in RELEASE.2019–04–04T18–31–46Z. Severity: High Who is affected: All users of the MinIO server are affected. Users of the MinIO gateway are not affected. However, it is still recommended to upgrade. Recommended Action for Users: All users

Read more...

Security Advisory

Synopsis: Possibility of spoofing authentication as another user on the Minio server S3 and Admin API was discovered and has been fixed in RELEASE.2019–02–20T22–44–29Z Severity: Medium Who is affected: All users using multi-user [https://docs.minio.io/docs/minio-multi-user-quickstart-guide.html] feature are affected. However, it is still recommended for everyone to upgrade. Recommended Action for

Read more...

Security Advisory

Synopsis: Possibility of authentication bypass against the Minio server Storage API was discovered and has been fixed in RELEASE.2019–02–12T21–58–47Z Severity: Critical Who is affected: The concerned issue is present in all the Minio releases after October 4th 2018. All users of distributed erasure backend are affected. Users of FS and Gateway backend are not affected.

Read more...

Security Advisory

Synopsis: A violation of the SSE-C security guarantees was discovered and has been fixed in RELEASE.2018–07–10T01–42–11Z [https://github.com/minio/minio/releases/tag/RELEASE.2018-07-10T01-42-11Z]. Severity: Low Who is affected: All users who stored objects using the S3 SSE-C API and used the same client-provided key at least twice for different objects. Recommended Action for

Read more...

Security Advisory

Synopsis: A Denial-of-Service (DoS) vulnerability against the Minio server was discovered and has been fixed in RELEASE.2018–05–25T19–49–13Z [https://github.com/minio/minio/tree/RELEASE.2018-05-25T19-49-13Z] Severity: Medium Who is affected: All users of the signature V4 authentication are affected. Users of the signature V2 authentication are not affected. Recommended Action for Users: All users are

Read more...

Here Comes The Flood

Here Comes The Flood

In 1994, EMC’s Symmetrix set an industry high-water mark with its max capacity of 1TB (TeraByte) for an enterprise storage system. In 2016, NetApp introduced the FAS9000 which in a 24-node cluster config holds about 172PB (PetaByte) raw. Increasing by 172,000x after 22 years — while dramatically improving performance and adding dozens of wonderful features — is decidedly impressive. But

Read more...

Experimental Amazon S3 API support for Backblaze B2 Cloud Storage Service

Some our community members have repeatedly asked for Backblaze B2 Cloud Storage and MinIO integration [1] [https://github.com/minio/minio/issues/4072] . B2 is competitively priced and has a huge fan following. We also heard from Backblaze team that they are actively expanding their B2 cloud storage service. We added experimental support for Backblaze B2 backend in MinIO to

Read more...

Go implementation of Data At Rest Encryption

Introduction Encrypting network traffic is becoming the default. There are standardized protocols like SSH and TLS as well as projects like Let’s Encrypt to protect data sent over the network. TLS for example takes a data stream, chunks the stream into messages and encrypts every message before sending it through the network. TLS ensures that each message is encrypted

Read more...

MinIO server gauge with Raspberry Pi

Today we’ll take a look at an IoT pressure gauge for your MinIO server. This is a cool hack built by Alex Ellis with the Raspberry Pi, Docker and MinIO. > After contributing code to the Open-Source MinIO project to enable webhooks Alex wanted visualize the traffic passing through his server. His hack shows the rate that objects get

Read more...

Debugging Go Routine leaks

Before beginning with Debugging Goroutines leaks, let me give a small introduction of some fundamentals which will give you a broader perspective of the problem. Concurrent Programming. * Concurrent programming deals with concurrent execution of a program wherein multiple sequential streams of execution run at a time resulting in faster execution of your computations. * It helps in better utilization of multicore

Read more...

How to use VMware Harbor with MinIO?

Harbor [https://github.com/vmware/harbor] is an enterprise-class docker registry server to store and distribute container images. Follow this document to use MinIO object storage server as a storage backend for Harbor container registry. Prerequisites Install and run MinIO [https://github.com/minio/minio#docker-container] server docker run -p 9000:9000 --name minio \ -e "MINIO_ACCESS_KEY=minio&

Read more...

Publish Minio events via NATS

Introduction Minio [https://www.minio.io/] server supports Amazon S3 compatible bucket event notification for following targets AMQP [https://www.amqp.org/about/what], Elasticsearch [https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html] , Redis [http://redis.io/documentation], nats.io [http://nats.io/], PostgreSQL [https://www.postgresql.org/] and Apache Kafka [http://kafka.apache.org/]. Part 4

Read more...

Publish MinIO events via Redis

Introduction Minio [https://www.minio.io/] server supports Amazon S3 compatible bucket event notification for following targets AMQP [https://www.amqp.org/about/what], Elasticsearch [https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html] , Redis [http://redis.io/documentation], nats.io [http://nats.io/], PostgreSQL [https://www.postgresql.org/] and Apache Kafka [http://kafka.apache.org/]. Part 3

Read more...