Frank Wessels - MinIO Blog

Bringing ARM into the AI Data Infrastructure Fold at MinIO Using SVE

Frank Wessels Frank Wessels on AI/ML 22 July 2024

One of the reasons that MinIO is so performant is that we do the granular work that others will not or cannot. From SIMD acceleration to the AVX-512 optimizations we have done the hard stuff. Recent developments for the ARM CPU architecture, in particular Scalable Vector Extensions (SVE), presented us with the opportunity to deliver significant performance and efficiency gains

Simplicity at Scale

Frank Wessels Frank Wessels on 20 June 2022

Frank Wessels is a founder at Sneller and previously worked at MinIO. Introduction While at MinIO, perhaps the most important lesson that I learned was about simplicity. Simplicity is is paramount if you want to achieve (true) scalability. This permeated MinIO's architecture and software design from front to back and drove, amongst others, the important Sneller decisions that

Impact of Intel vs. ARM CPU Performance for Object Storage

Frank Wessels Frank Wessels on Intel 3 June 2020

The recent announcement from AWS about the general availability of their new ARM-powered Graviton2 servers caused us to take another look at the performance of these ARM servers. In this blog post we describe the results which you may find surprising. Introduction MinIO [https://github.com/minio/minio] is an Apache licensed, open source S3-compatible object storage server with a

Accelerating aggregate MD5 hashing up to 800% with AVX512

Frank Wessels Frank Wessels on Performance 28 April 2020

Introduction While MD5 hashing is no longer a good choice when considering a hash function, it is still being used in a great variety of applications. As such any performance improvements that can be made to the MD5 hashing speed are worth considering. Due to recent improvements in SIMD processing (AVX2 and especially AVX512) we are providing a Go md5-simd

simdjson-go: Parsing gigabytes of JSON per second in Go

Frank Wessels Frank Wessels on Golang 10 February 2020

Introduction JSON has established itself as the "lingua franca" of the web. As such the parsing performance of JSON is hugely important for many applications. Despite the simple and human-friendly nature of JSON, it is not a technically trivial format to parse at high speeds. Recently some new designs have been presented one of which is simdjson [https:

Running Peta-Scale Spark Jobs on Object Storage Using S3 Select

Frank Wessels Frank Wessels on Performance 12 May 2019

When one looks at the amazing roster of talks for the Spark + AI Summit, what you don’t see is a lot of discussion on how to leverage object storage. On some level you would expect to —

Golang Internals Part 2: Nice benefits of named return values

Frank Wessels Frank Wessels on Golang 28 June 2017

You may know that Golang offers the ability to name return values. Thus far at MinIO [https://github.com/minio/minio] we have not been using this feature much, but that’ll change since there are some nice hidden benefits as we will explain in this blog post. If you are like us, you may have considerable amounts of code

Golang Internals Part 1: Autogenerated functions (and how to get rid of them)

Frank Wessels Frank Wessels on Golang 14 June 2017

Maybe if you are like us at Min [https://github.com/minio/minio]IO, you have every now and then come across ‘autogenerated’ functions in your Golang call stacks and wondered what they are all about? We had a case the other day where the call stack showed something like the following: cmd.retryStorage.ListDir(0x12847c0, 0xc420402e70, 0x1, ...) minio/cmd/

minio/dsync: a distributed locking and syncing package for Go

Frank Wessels Frank Wessels on Performance 15 September 2016

minio/dsync [https://github.com/minio/dsync] is a package for doing distributed locks over a network of nnodes. It is designed with simplicity in mind and offers limited scalability (n <= 16). Each node is connected to all other nodes and lock requests from any node will be broadcast to all connected nodes. A node will succeed in getting

Accelerating SHA256 by 100x in Golang on ARM

Frank Wessels Frank Wessels on Performance 26 July 2016

The 64-bit ARMv8 core has introduced new instructions for SHA1 and SHA2 acceleration as part of the Cryptography Extensions [http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0501f/CHDFJBCJ.html] . We at Minio [https://minio.io/] were curious as to the difference that these instructions might make, and this turns out to be one of the nicer

Fast hashing in Golang using BLAKE2

Frank Wessels Frank Wessels on Performance 6 July 2016

In this blog we would like to present an optimized implementation, blake2b-simd [https://github.com/minio/blake2b-simd] , in pure Go for the BLAKE2 [https://blake2.net/] hashing algorithm that takes advantage of SIMD instructions. It gives up to a 4x speed increase over the (non-assembly) Go implementation and can achieve hashing speeds close to 1 GB/sec per core on