One of the reasons that MinIO is so performant is that we do the granular work that others will not or cannot. From SIMD acceleration to the AVX-512 optimizations we have done the hard stuff. Recent developments for the ARM CPU architecture, in particular Scalable Vector Extensions (SVE), presented us with the opportunity to deliver significant performance and efficiency gains
Read more
Frank Wessels is a founder at Sneller and previously worked at MinIO. Introduction While at MinIO, perhaps the most important lesson that I learned was about simplicity. Simplicity is is paramount if you want to achieve (true) scalability. This permeated MinIO's architecture and software design from front to back and drove, amongst others, the important Sneller decisions that
Read more
The recent announcement from AWS about the general availability of their new ARM-powered Graviton2 servers caused us to take another look at the performance of these ARM servers. In this blog post we describe the results which you may find surprising. Introduction MinIO [https://github.com/minio/minio] is an Apache licensed, open source S3-compatible object storage server with a
Read moreIntroduction While MD5 hashing is no longer a good choice when considering a hash function, it is still being used in a great variety of applications. As such any performance improvements that can be made to the MD5 hashing speed are worth considering. Due to recent improvements in SIMD processing (AVX2 and especially AVX512) we are providing a Go md5-simd
Read more
Introduction JSON has established itself as the "lingua franca" of the web. As such the parsing performance of JSON is hugely important for many applications. Despite the simple and human-friendly nature of JSON, it is not a technically trivial format to parse at high speeds. Recently some new designs have been presented one of which is simdjson by
Read more
When one looks at the amazing roster of talks for the Spark + AI Summit, what you don’t see is a lot of discussion on how to leverage object storage. On some level you would expect to —
Read moreYou may know that Golang offers the ability to name return values. Thus far at MinIO [https://github.com/minio/minio] we have not been using this feature much, but that’ll change since there are some nice hidden benefits as we will explain in this blog post. If you are like us, you may have considerable amounts of code
Read moreMaybe if you are like us at Min [https://github.com/minio/minio]IO, you have every now and then come across ‘autogenerated’ functions in your Golang call stacks and wondered what they are all about? We had a case the other day where the call stack showed something like the following: cmd.retryStorage.ListDir(0x12847c0, 0xc420402e70, 0x1, ...) minio/cmd/
Read moreminio/dsync [https://github.com/minio/dsync] is a package for doing distributed locks over a network of nnodes. It is designed with simplicity in mind and offers limited scalability (n <= 16). Each node is connected to all other nodes and lock requests from any node will be broadcast to all connected nodes. A node will succeed in getting
Read more
The 64-bit ARMv8 core has introduced new instructions for SHA1 and SHA2 acceleration as part of the Cryptography Extensions [http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0501f/CHDFJBCJ.html] . We at Minio [https://minio.io/] were curious as to the difference that these instructions might make, and this turns out to be one of the nicer
Read moreIn this blog we would like to present an optimized implementation, blake2b-simd [https://github.com/minio/blake2b-simd] , in pure Go for the BLAKE2 [https://blake2.net/] hashing algorithm that takes advantage of SIMD instructions. It gives up to a 4x speed increase over the (non-assembly) Go implementation and can achieve hashing speeds close to 1 GB/sec per core on
Read more