The growth of Presto in the enterprise is a function of its speed, SQL compatibility, extensibility and enterprise feature set. While initially designed to speed up Hadoop, the success of the project has led to much broader adoption - on S3, Cassandra, MySQL, and more. Presto allows for data queries that traverse data stores and locations - a big plus
Read more...
Well written software is fast software. When MinIO was conceived it was designed from scratch to be simple, to scale (because simple things scale better) and to be fast. Simplicity and scale have their own subjective and objective measures - but fast is generally a numbers game. When you take well-written, fast software and pair it with fast hardware the
Read more...
High performance object storage is one of the hotter topics in the enterprise today. On the one hand, object storage has become an indispensable part of the enterprise storage strategy (public or private cloud) - carrying the vast, vast majority of the enterprise burden when measured in TBs or PBs. On the other hand, object storage has traditionally served a
Read more...
Software isn't usually described as bombproof. Particularly the type of software that is responsible for large analytic jobs or machine learning workloads. The words “finicky”, “complex” or in the case of good marketing “professional grade” (meaning you need years of study and multiple certifications) are more common. Bombproof software, however, is one of the many benefits associated with
Read more...
When one looks at the amazing roster of talks for the Spark + AI Summit, what you don’t see is a lot of discussion on how to leverage object storage. On some level you would expect to —
Read more...
One of the key challenges in any digitization journey is the adoption of machine learning techniques. Given the explosion of tools and frameworks, it can be difficult
Read more...
When early object storage APIs were developed they focused on the efficient storage and retrieval of objects. Amazon’s success with S3 and its implementation of the robust S3 API quickly became the de facto standard for object storage in the cloud. MinIO, recognizing this, invested heavily in creating the most compliant implementation of the S3 API outside of Amazon.
Read more...
How to use Apache Flink to build a private cloud data pipeline for a variety of use cases.
Read more...
In this post we’ll learn more about object storage, specifically Minio and then see how to connect Minio with tools like Apache Spark and Presto for analytics workloads.
Read more...
In the first part of this two post series, we’ll take a look at how object storage is different from other storage approaches and why it makes sense to leverage object storage like Minio for data lakes.
Read more...
In this post, we learn about Pivotal Container Service deployment and how to use the pks command line tool to create and manage Kubernetes clusters. We also saw how to deploy Minio once your PKS Kubernetes cluster is set up and running.
Read more...
In this post, we learn about why and how Presto is becoming the tool of choice when querying large datasets from platforms like MinIO. We then learn the steps to setup and deploy Presto on private infrastructure.
Read more...
Minio now has built-in support for exporting Prometheus compatible data on an unauthenticated endpoint. This enables Prometheus monitoring for Minio server deployments without sharing server credentials and eliminates the need to run an external Prometheus exporter.
Read more...There has been a lot going on MinIO server development front. We recently added support for * Disk Caching [https://docs.minio.io/docs/minio-disk-cache-guide] * Large/Petascale buckets [https://docs.minio.io/docs/minio-large-bucket-support-quickstart-guide] * Storage Classes [https://github.com/minio/minio/tree/master/docs/erasure/storage-class] While features like MinIO bucket federation [https://github.com/minio/minio/pull/5501], revamped MinIO
Read more...ZFS is best known for abstracting away the physical storage device boundaries by pooling them together. ZFS completely removed the need to manually handle physical storage or worry about their individual capacities. ZFS is also a pioneer in its ability to detect data corruption and recover if data redundancy is available. However, as we already discussed [https://blog.minio.io/
Read more...
One of the major requirements for success with IoT strategy is the ability to store and analyze device and sensor data. As IoT brings thousands of devices online everyday, the data being generated by all these devices combined is reaching staggering levels. > Storing the IoT data in a scalable yet cost effective manner, while being able to analyze it
Read more...
Erasure coding [https://en.wikipedia.org/wiki/Erasure_code] is a reliable approach to shard data across multiple drives and fetch it back, even when a few of the drives are not available. MinIO server leverages Reed Solomon [https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction] erasure coding to ensure data redundancy in multiple disk deployments. Until
Read more...
We take a look at COSBench performance numbers with Minio server running on NVMe drives
Read more...Before beginning with Debugging Goroutines leaks, let me give a small introduction of some fundamentals which will give you a broader perspective of the problem. Concurrent Programming. * Concurrent programming deals with concurrent execution of a program wherein multiple sequential streams of execution run at a time resulting in faster execution of your computations. * It helps in better utilization of multicore
Read more...minio/dsync [https://github.com/minio/dsync] is a package for doing distributed locks over a network of nnodes. It is designed with simplicity in mind and offers limited scalability (n <= 16). Each node is connected to all other nodes and lock requests from any node will be broadcast to all connected nodes. A node will succeed in getting
Read more...
Erasure Code Calculator
Reference Hardware
