In this post, we learn about why and how Presto is becoming the tool of choice when querying large datasets from platforms like MinIO. We then learn the steps to setup and deploy Presto on private infrastructure.
Read more...
Minio now has built-in support for exporting Prometheus compatible data on an unauthenticated endpoint. This enables Prometheus monitoring for Minio server deployments without sharing server credentials and eliminates the need to run an external Prometheus exporter.
Read more...There has been a lot going on MinIO server development front. We recently added support for * Disk Caching [https://docs.minio.io/docs/minio-disk-cache-guide] * Large/Petascale buckets [https://docs.minio.io/docs/minio-large-bucket-support-quickstart-guide] * Storage Classes [https://github.com/minio/minio/tree/master/docs/erasure/storage-class] While features like MinIO bucket federation [https://github.com/minio/minio/pull/5501], revamped MinIO
Read more...ZFS is best known for abstracting away the physical storage device boundaries by pooling them together. ZFS completely removed the need to manually handle physical storage or worry about their individual capacities. ZFS is also a pioneer in its ability to detect data corruption and recover if data redundancy is available. However, as we already discussed [https://blog.minio.io/
Read more...
One of the major requirements for success with IoT strategy is the ability to store and analyze device and sensor data. As IoT brings thousands of devices online everyday, the data being generated by all these devices combined is reaching staggering levels. > Storing the IoT data in a scalable yet cost effective manner, while being able to analyze it easily
Read more...
Erasure coding [https://en.wikipedia.org/wiki/Erasure_code] is a reliable approach to shard data across multiple drives and fetch it back, even when a few of the drives are not available. MinIO server leverages Reed Solomon [https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction] erasure coding to ensure data redundancy in multiple disk deployments. Until
Read more...
We take a look at COSBench performance numbers with Minio server running on NVMe drives
Read more...Before beginning with Debugging Goroutines leaks, let me give a small introduction of some fundamentals which will give you a broader perspective of the problem. Concurrent Programming. * Concurrent programming deals with concurrent execution of a program wherein multiple sequential streams of execution run at a time resulting in faster execution of your computations. * It helps in better utilization of multicore
Read more...minio/dsync [https://github.com/minio/dsync] is a package for doing distributed locks over a network of nnodes. It is designed with simplicity in mind and offers limited scalability (n <= 16). Each node is connected to all other nodes and lock requests from any node will be broadcast to all connected nodes. A node will succeed in getting the
Read more...
The 64-bit ARMv8 core has introduced new instructions for SHA1 and SHA2 acceleration as part of the Cryptography Extensions [http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0501f/CHDFJBCJ.html] . We at Minio [https://minio.io/] were curious as to the difference that these instructions might make, and this turns out to be one of the nicer
Read more...In this blog we would like to present an optimized implementation, blake2b-simd [https://github.com/minio/blake2b-simd] , in pure Go for the BLAKE2 [https://blake2.net/] hashing algorithm that takes advantage of SIMD instructions. It gives up to a 4x speed increase over the (non-assembly) Go implementation and can achieve hashing speeds close to 1 GB/sec per core on
Read more...
In my previous post [https://blog.minio.io/object-storage-what-is-it-all-about-62920ca164ca#.bcz4d4nnd] we learnt the why and what of object storage. Specifically, we learnt why at all a new storage paradigm is required and what it does to alleviate modern unstructured data problems. We saw how object storage lets you access objects at application layer with simple API calls over HTTP(s)
Read more...
on
Performance
One of the fundamental requirements of microservice is that application containers become stateless. However the states still need to be stored elsewhere, as in databases, object storage, session files, cookies, cache, etc. Microstorage is emerging as a new architecture to address the storage scalability for microservices. Microstorage inherits the idea from microservices that “only small things scale”. Scale-up is not
Read more...
Hybrid Cloud Object Storage
Baremetal
Erasure Code Calculator
Reference Hardware